diff --git a/Cargo.lock b/Cargo.lock index 8254a20b071..36be1501e49 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -5757,6 +5757,7 @@ name = "moq-tokio" version = "0.19.23" dependencies = [ "anyhow", + "bench-harness", "bytes", "chrono", "dns-lookup", @@ -8036,7 +8037,6 @@ dependencies = [ "moq-quic", "qmux 0.6.5", "rcgen 0.14.10", - "rustc-hash 2.1.3", "rustls", "thiserror 2.0.21", "tokio", diff --git a/quest/m1/quic-api/README.md b/quest/m1/quic-api/README.md index 4c0dea2bdf6..25691a17e26 100644 --- a/quest/m1/quic-api/README.md +++ b/quest/m1/quic-api/README.md @@ -36,7 +36,7 @@ Decided with the maintainer: WebTransport into m1 from m2, and deleted 0-RTT entirely: MoQ cannot use it, since CONNECT and authorization need the finished handshake. - 2026-10-25: moq-quic owns every wait as a `poll_*` on `&kio::Waiter` - ([waits](/quest/m1/quic-api/waits.md)), and moq-net's `transport::poll` + (landed as the waits quest), and moq-net's `transport::poll` traits join the epic so all three layers share the `&self, &kio::Waiter` shape. @@ -46,7 +46,6 @@ side by side and fix any name or signature that still differs, and check ## Required -- [Waits](/quest/m1/quic-api/waits.md) - moq-quic owns every wait as a `poll_*` on `&kio::Waiter`, deleting `StreamEvent` and the drivers' and qmux's waiter tables - [Poll forms](/quest/m1/quic-api/poll.md) - both drivers' waits poll as `&self, &kio::Waiter` forwards to moq-quic, the async forms wrap them, and `send::Finished` is a future - [Native WebTransport](/quest/m1/quic-api/native-web.md) - `quic::web` in both crates implements moq-net's poll traits, deleting the generic `web_transport_trait` adapters and the dependency on it diff --git a/quest/m1/quic-api/poll.md b/quest/m1/quic-api/poll.md index cd7604c7c40..75c25623500 100644 --- a/quest/m1/quic-api/poll.md +++ b/quest/m1/quic-api/poll.md @@ -30,7 +30,7 @@ primary surface. [Native WebTransport](/quest/m1/quic-api/native-web.md) and Decided 2026-10-25 (maintainer), replacing the 2026-10-23 shape: -- **moq-quic owns the waits** ([waits](/quest/m1/quic-api/waits.md)). Each +- **moq-quic owns the waits** (landed: `connection/waits.rs`). Each connection and stream form here takes the lock and forwards to moq-quic's `poll_*` in one line; the driver keeps no waiter table of its own. Only `Endpoint::poll_accept` waits on driver state, the queue of handshaken @@ -69,7 +69,7 @@ async forms unchanged. Added 2026-11-01 (maintainer, follow-up planning after the 2026-11-01 /quest-afk run), from [moq-dev/moq#7490](https://github.com/moq-dev/moq/pull/7490), -which finishes [waits](/quest/m1/quic-api/waits.md): +which finishes the waits quest: - **Remove `ShouldTransmit`.** Once moq-quic wakes the transmit waiter itself, the `ShouldTransmit` that `Chunks::finalize` and the send lease methods @@ -86,10 +86,6 @@ which finishes [waits](/quest/m1/quic-api/waits.md): a test that a close behind it is still bounded, mirroring qmux's `a_close_behind_a_stalled_peer_is_bounded`. -## Required - -- [Waits](/quest/m1/quic-api/waits.md) - moq-quic's `poll_*` forms these forward to - ## Related - [Native WebTransport](/quest/m1/quic-api/native-web.md) - its native QUIC `poll::Session` calls these diff --git a/quest/m1/quic-api/waits.md b/quest/m1/quic-api/waits.md deleted file mode 100644 index 7ac3ffbd3cf..00000000000 --- a/quest/m1/quic-api/waits.md +++ /dev/null @@ -1,100 +0,0 @@ -# [L] moq-quic owns its waits - -## Goal - -moq-quic's connection, streams, and datagrams expose every wait as a -`poll_*` taking `&kio::Waiter`, and its own state wakes the waiters as -`handle`, timeouts, and application calls change it. `StreamEvent` and the -waits in the connection's `Event` are gone, and moq-tokio's -`quic::driver`, moq-uring's driver, and qmux delete their per-stream and -per-connection waiter tables, keeping only I/O, timers, and their lock. -moq-quic still does no I/O and reads no clock. - -## Plan - -Landed in [moq-dev/moq#7391](https://github.com/moq-dev/moq/pull/7391): moq-quic -owns the lists and wakes them, over QUIC and QMux alike, alongside the events. -`connection/waits.rs` holds them, owned by `StreamsState`; `StreamsState::notify` -wakes the list matching each `StreamEvent` it queues, and `StreamsState::unblock` -wakes connection-blocked writers as soon as the window has room rather than when -`poll` runs. The poll forms are `recv::Chunks::poll_next`, -`recv::Stream::poll_lease`, `send::Stream::poll_write`, `poll_end`, and -`poll_acked`, `Streams::poll_accept` and `poll_open`, `Datagrams::poll_recv`, and -`Connection::poll_connected`, `poll_handshake_confirmed`, and `poll_closed`, with -`ConnectionLost` on `send::Error` and `recv::Error` for a stream wait that -outlives its connection. Counting-waiter tests cover each wait, two waiters on -one condition, a dropped waiter, and draining after a loss. - -What remains: - -- Move moq-tokio's `quic::driver`, moq-uring's driver, and qmux onto the poll - forms, deleting their waiter tables, `Notify`s, and `stopped` map. -- Delete `StreamEvent` and the waiting `Event` variants, turning `notify`'s - event-to-list mapping into direct wakes, and move each `Event` test that - asserted a wait to a counting waiter. -- Make `poll_transmit` the driver's wait, below. -- The close-while-output-blocked qmux test and the benches, below. - -Decided with the maintainer on 2026-10-25, from "would moq-quic benefit from -kio?": - -- **The core owns the waits.** Today moq-quic reports - `StreamEvent::{Opened, Readable, Writable, Finished, Acked}` and - connection `Event`s, and each consumer keeps its own tables - (`blocked_readers`/`blocked_writers` and `Notify`s in moq-tokio, - `readable`/`writable`/`finishing`/accept/open/datagram `WaiterList`s in - moq-uring and qmux), about 25 wake sites each, and looks every stream id - up a second time. The waiter lists move into moq-quic's per-stream and - per-connection state instead. s2n-quic's transport is prior art: wakers in - stream state, I/O in pluggable providers. -- **`&kio::Waiter`, the `rs/AGENTS.md` convention.** kio is already public - through moq-net (`pub use kio`), so a kio break already breaks the stack. - The caller owns the waiter, so an abandoned poll leaves nothing behind. -- **Every wait.** Accept (per direction), open (stream credit), read, write, - finished (FIN acknowledged or STOP_SENDING), acked (the - [poll_acked](/quest/m2/quic-ack-hook.md) offset wait), datagram receive, - connected, handshake confirmed, and closed. `Event` keeps only - notifications no handle waits on, or goes if none remain. Endpoint-level - plumbing (`EndpointEvent`, `ConnectionEvent`) is not a wait and stays. -- **`poll_transmit` is the driver's wait.** It takes the driver's waiter and - returns `Poll`: with nothing sendable it registers the waiter, - which a write, open, reset, stop, close, or credit change wakes. Pacing - and loss timers stay on `poll_timeout`. This replaces the kick each - handle sends its driver after mutating the connection. -- **After [moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)**, whose queryable - send-stream end is what `poll_finished` and `poll_acked` read, and whose - names the new methods take. - -Constraints: - -- A wake only schedules. moq-quic wakes under the consumer's lock (the - tokio driver's `Mutex`, moq-uring's and qmux's `RefCell`), as the drivers - do today, so a waker that polled inline would re-enter it. -- A driver stalled on output still hears a local close. `poll_transmit` - registers only when nothing is sendable, but qmux's `poll_stalled` - parks on its `driver` list while a batch is blocked in `feed`/`flush`, and - a local close wakes it to start the bounded close timer. Keep that through - a wait that consumes no transmit (a close wait or a non-consuming - readiness wait), and test it on paused time: close while output is - blocked by a non-reading peer, and the session still shuts down within - the close bound. -- Waits drain before failing: accepted streams, readable data, and received - datagrams stay available after a close until taken, then the error. -- Tests in moq-quic count wakes with a counting waiter instead of matching - events: two waiters on one condition both wake, a dropped waiter leaves no - live registration, and each `Event` test that asserted a wait moves over. -- Measure: `rs/moq-quic/benches/streams.rs` and a driver fanout bench - before and after, swept over streams per connection, so the per-event cost - shows as a slope if the lists cost more than the maps they replace. - -Public API: moq-quic gains `poll_*` methods taking `&kio::Waiter`, -`poll_transmit` changes shape, and `StreamEvent` and the waiting `Event` -variants go. The drivers' and qmux's public APIs are unchanged here; the -driver poll forms come in [poll](/quest/m1/quic-api/poll.md). Update -moq-quic's README fork list and describe migration in the PR's Impact section. Wire: none. - -## Related - -- [Poll forms](/quest/m1/quic-api/poll.md) - the drivers' public poll forms forward to these -- [poll_acked](/quest/m2/quic-ack-hook.md) - its offset wait lives here -- [QUIC lock contention](/quest/m2/perf/quic-lock-contention.md) - profiles the lock these waits are registered under diff --git a/quest/m2/perf/quic-lock-contention.md b/quest/m2/perf/quic-lock-contention.md index fd768bece7d..8996e7e750f 100644 --- a/quest/m2/perf/quic-lock-contention.md +++ b/quest/m2/perf/quic-lock-contention.md @@ -43,8 +43,8 @@ Levers, only where the profile shows a cost: with [the I/O boundary redesign](/quest/m2/quic-io-boundary.md), whose "one `handle` call takes a whole GRO batch" is the state-machine half. - Stream handles that carry a slab key instead of a `StreamId`, removing - moq-quic's per-stream lookups (the driver's own maps go with - [waits](/quest/m1/quic-api/waits.md)). FxHash is cheap, so this is last + moq-quic's per-stream lookups (the driver's own maps are already gone, + now that moq-quic owns its waits). FxHash is cheap, so this is last and only on evidence. The maintainer wants the connection state free of runtime borrow checks @@ -65,7 +65,6 @@ staging queue and batch dispatch should need none. ## Required - [Native WebTransport](/quest/m1/quic-api/native-web.md) - removes the adapter's per-operation boxing, which would otherwise dominate the profile -- [Waits](/quest/m1/quic-api/waits.md) - moves the waiter tables under the lock this profiles ## Related diff --git a/quest/m2/quic-ack-hook.md b/quest/m2/quic-ack-hook.md index 6e4b81a6d64..76775e82514 100644 --- a/quest/m2/quic-ack-hook.md +++ b/quest/m2/quic-ack-hook.md @@ -53,9 +53,8 @@ The same pair goes on `send::Finished`, the handle `finish(self)` returns ([moq-dev/moq#6950](https://github.com/moq-dev/moq/pull/6950)), and the methods take its opaque `send::Offset`, which only `offset()` mints, instead of a `u64`. The `send::Error` says why the bytes will never be acknowledged: a reset, the peer's STOP_SENDING, or the connection's -loss. Both drivers forward to `moq-quic`'s own `poll_acked` wait, which -[waits](/quest/m1/quic-api/waits.md) makes of today's `acked(offset)` and -`StreamEvent::Acked`. Four traps, each settled in moq-quic where the wait +loss. Both drivers forward to `moq-quic`'s own `send::Stream::poll_acked` +wait, which replaced `StreamEvent::Acked`. Four traps, each settled in moq-quic where the wait lives: - A finished stream whose every byte was acknowledged must not look like a diff --git a/quest/m2/quic-sansio-driver.md b/quest/m2/quic-sansio-driver.md index 5b83430074a..c55c9a31fd8 100644 --- a/quest/m2/quic-sansio-driver.md +++ b/quest/m2/quic-sansio-driver.md @@ -18,7 +18,7 @@ whose C, C++, and Kotlin bindings drive it from their own loops Decided with the maintainer on 2026-10-28, from [moq-dev/rust2#215](https://github.com/moq-dev/rust2/pull/215): -- **After [waits](/quest/m1/quic-api/waits.md) and [poll](/quest/m1/quic-api/poll.md).** +- **After the waits (landed) and [poll](/quest/m1/quic-api/poll.md).** Once moq-quic owns its waits, a driver is I/O, timers, and a lock. This one is the timers and the lock, with the caller doing the I/O. Built earlier, it would carry the waiter tables that waits deletes. @@ -49,5 +49,4 @@ impl. ## Required -- [Waits](/quest/m1/quic-api/waits.md) - moq-quic owns every wait, so the driver keeps no waiter table - [Poll forms](/quest/m1/quic-api/poll.md) - the drivers' poll shape this one mirrors diff --git a/rs/moq-quic/MIGRATE.md b/rs/moq-quic/MIGRATE.md index 9327fc8dbda..61908a96f45 100644 --- a/rs/moq-quic/MIGRATE.md +++ b/rs/moq-quic/MIGRATE.md @@ -4,6 +4,32 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Waits replace stream and connection events + +Every wait is a `poll_*` taking the caller's `&kio::Waiter`, which moq-quic registers and wakes itself, so a driver keeps no waiter tables. +`StreamEvent` is gone, and so are the `Event` variants a handle waited on; `Event` keeps only `PathUpdated`, and `mux::Event` and `mux::Connection::poll` are gone. + +| Before | After | +| --- | --- | +| `Event::HandshakeDataReady` | `Connection::poll_handshake_data` | +| `Event::Connected` | `Connection::poll_connected` | +| `Event::HandshakeConfirmed` | `Connection::poll_handshake_confirmed` | +| `Event::ConnectionLost { reason }` | `Connection::poll_closed`, or `Connection::error` | +| `Event::DatagramReceived` | `Datagrams::poll_recv` | +| `StreamEvent::Opened { dir }` | `Streams::poll_accept` | +| `StreamEvent::Available { dir }` | `Streams::poll_open` | +| `StreamEvent::Readable { id }` | `recv::Chunks::poll_next`, `recv::Stream::poll_lease`, or `recv::Stream::poll_received_reset` | +| `StreamEvent::Writable { id }` | `send::Stream::poll_write` | +| `StreamEvent::Finished { id }`, `StreamEvent::Stopped { id, .. }` | `send::Stream::poll_end` | +| `StreamEvent::Acked { id, .. }` | `send::Stream::poll_acked` | + +`poll_transmit` is the driver's wait, so a handle no longer kicks its driver after a call. +`Connection::poll_transmit(now, max_datagrams, buf) -> Option` becomes `poll_transmit(waiter, now, max_datagrams, buf) -> Poll`, and `mux::Connection::poll_transmit(now) -> Option` becomes `poll_transmit(waiter, now) -> Poll`. +With nothing to send it registers `waiter`, which the next call that queues something wakes; keep waiting on `poll_timeout` for pacing and loss recovery. + +A wake only schedules: moq-quic wakes under whatever lock guards the connection, so the woken task polls again under it. +Stream waits fail with `send::Error::ConnectionLost` or `recv::Error::ConnectionLost` once the connection is lost, after accepted streams, readable bytes, and received datagrams drain. + ### Crypto is sealed to rustls ([#7028](https://github.com/moq-dev/moq/pull/7028)) Build configs with `client::Config::new(Arc::new(rustls_client))` and `server::Config::new(Arc::new(rustls_server))`; both are infallible. @@ -24,7 +50,7 @@ Treat it as a failed read and stop the stream: a reader that watches for a reset ### Early data is removed -Wait for `Event::Connected` before opening streams or sending application data. +Wait for `Connection::poll_connected` before opening streams or sending application data. `Connection::has_0rtt`, `accepted_0rtt`, and the `crypto::Session::early_crypto` and `early_data_accepted` hooks are gone. Custom crypto sessions no longer implement those hooks. Rustls configs disable early data, including caller-supplied configs, while session tickets still resume 1-RTT handshakes. diff --git a/rs/moq-quic/README.md b/rs/moq-quic/README.md index e532ec525d1..f03a05e3813 100644 --- a/rs/moq-quic/README.md +++ b/rs/moq-quic/README.md @@ -52,7 +52,7 @@ Changes on top of the upstream commit, besides the renames: - **QUIC v1 only**: `Version` names the one version spoken; drafts 29 to 34, their initial salts, and their retry keys are gone, since no MoQ peer speaks them. - **Plain configs**: `transport`, `client`, `server`, and `endpoint` configs are structs with public fields instead of setters, validated when an endpoint is built or dials. `Limits` is the one spelling of the windows and stream count, at construction and on a live connection, where it also changes the stream receive window. -- **Close events**: a local close reports `connection::Error::LocallyClosed` with its code and reason as the connection's `ConnectionLost`, a handshake that idles out reports `HandshakeTimeout`, and close reasons are strings, lossily decoded. +- **Close reasons**: a local close reports `connection::Error::LocallyClosed` with its code and reason from `poll_closed` and `error()`, a handshake that idles out reports `HandshakeTimeout`, and close reasons are strings, lossily decoded. - **No early data**: TLS never offers or accepts 0-RTT, and early packets are discarded without decryption. The early keys, remembered transport parameters, stream rollback, and early-session driver APIs are removed. Session tickets still resume 1-RTT handshakes. MoQ's CONNECT and authorization already wait for the handshake. - **BBR3**, from [quinn#2481](https://github.com/quinn-rs/quinn/pull/2481) at [`55f74c0d`](https://github.com/quinn-rs/quinn/pull/2481/commits/55f74c0dd0738b5ad7d7676c0b2869a547783f83) (open upstream), rebased onto the fork point with authorship kept. It replaces quinn's BBR and is the default controller. It carries the BBR correctness fixes moq-dev/noq shipped in `moq-noq` 1.3.1, and with them the `Controller` changes BBR needs: per-packet send and loss callbacks, `on_cwnd_limited`, `on_app_limited`, `on_ack_frequency_update`, and controller-owned pacing and burst size. `Metrics` rates are bytes per second. - **BBR3 upstream refresh**: Checked against the force-pushed [`01b8848d`](https://github.com/quinn-rs/quinn/pull/2481/commits/01b8848dcb321eb17ad4d88dd6f84f28f45a0d6a) on 2026-10-31: carry startup's per-round ACK aggregation maximum, restore the saved window once when recovery ends, preserve it through recovery into ProbeRTT, saturate probe-bound growth, and stop a probe at the previous loss bound before accelerating again on clean feedback. Spurious-loss undo also restores that probe history, and an idle ACK renews application-limited marking even when it crosses the old marker. The two-cycle bandwidth filter and transport-owned packet snapshots already cover its other fixes. Upstream's bandwidth metric conversion to bits/s does not apply: this fork's controller metrics use bytes/s. @@ -83,24 +83,24 @@ Changes on top of the upstream commit, besides the renames: - **Scheduled datagrams** (`Datagrams::send`): a datagram is queued by its `send::Options`, in a send group (or ungrouped) at an `i64` priority, and scheduled with the group's streams, winning a tie with a stream of equal priority and spending the group's byte credit, instead of quinn's datagram-first FIFO in `populate_packet`. Sending never blocks: `datagram_send_buffer_size` is shared, and an overflow evicts the oldest datagram of the lowest-priority group. `SendDatagramError::Blocked` and `Event::DatagramsUnblocked` are gone, and a port touching the `// DATAGRAM` loop has to be redone against `StreamsState::write_frames`. - **Sharded connection IDs** (`RandomConnectionIdGenerator::set_shard`): every CID names shard `index` of a `count`-endpoint group in its first byte, modulo `count`, and sets a marker in its fourth byte, so a `SO_REUSEPORT` steering filter delivers a packet to the endpoint that issued its CID. `validate` refuses a CID without this shard's prefix and marker, so a misrouted packet is dropped instead of answered with a stateless reset. Not offered upstream; it serves moq-sock's per-core sharding. - **`RESET_STREAM_AT`** ([draft-ietf-quic-reliable-stream-reset](https://datatracker.ietf.org/doc/draft-ietf-quic-reliable-stream-reset/)): the `reset_stream_at` transport parameter is always advertised, `send::Stream::reset_at` keeps the data written before an offset reliable, holding a plain `RESET_STREAM` until that data is acknowledged when the peer lacks the extension, and a received reset surfaces only once the reliable prefix has been read. Stopping a stream after it was reset no longer credits its unread data twice. -- **ACK progress** (`send::Stream::offset`, `send::Stream::acked`, `StreamEvent::Acked`): `acked` reports whether the peer acknowledged every byte before an offset, by comparing against the stream's contiguous acknowledged prefix. A `false` stores the lowest polled offset on the stream, and the ACK that advances the prefix past it, or a reset or `STOP_SENDING` that makes it fail, reports `StreamEvent::Acked` with the prefix, so a port touching `StreamsState::received_ack_of`, `received_stop_sending`, or the reset path has to keep that report. Not offered upstream: quinn has no API for when sent data was acknowledged. +- **ACK progress** (`send::Stream::offset`, `send::Stream::acked`, `poll_acked`): `acked` reports whether the peer acknowledged every byte before an offset, by comparing against the stream's contiguous acknowledged prefix, which a reset that frees the stream keeps with its end. A `false` stores the lowest polled offset on the stream, and the ACK that advances the prefix past it, or a reset or `STOP_SENDING` that makes it fail, wakes the stream's `poll_acked` waiters, so a port touching `StreamsState::received_ack_of`, `received_stop_sending`, or the reset path has to keep that wake. Not offered upstream: quinn has no API for when sent data was acknowledged. - **Fuzzing** runs through the hidden `fuzz` module behind a `fuzz` feature, with one-line shims in [`fuzz/`](fuzz/README.md): upstream's `packet`, `params`, `streamid`, and `streams` targets, plus `frame` and a connection op sequence. Upstream's `cfg(fuzzing)` `fuzzing` module and `arbitrary` feature are gone, so its fuzz patches port by hand. - **Draining endpoint event**: `EndpointEvent::is_draining` fires once per connection when it stops being active, before the drained event, so the driver's `Endpoint::wait_all_draining` can end a shutdown without sitting out every closing period. -- **Late stream limits**: peer transport parameters that raise a stream limit after an open was refused emit `StreamEvent::Available`, as `MAX_STREAMS` does. A server accepted before the rest of a multi-packet ClientHello (a post-quantum key share is enough) otherwise leaves a 0.5-RTT `open_uni` parked until the idle timeout. Not offered upstream yet. +- **Late stream limits**: peer transport parameters that raise a stream limit after an open was refused wake `Streams::poll_open`, as `MAX_STREAMS` does. A server accepted before the rest of a multi-packet ClientHello (a post-quantum key share is enough) otherwise leaves a 0.5-RTT `open_uni` parked until the idle timeout. Not offered upstream yet. - **Receive leases** (`streams/lease.rs`, `recv::Stream::lease`), building on [#5675](https://github.com/moq-dev/moq/pull/5675)'s held credit: a lease reads a stream's ordered bytes out of the assembler but keeps their connection credit, counted in `StreamsState::held`, until the reader `advance`s or `take`s past them and `release`s, while stream credit advances as usual. A lease copies the bytes it reads out of their datagram buffers (`Leased::extend`), so a byte of credit never pins a receive batch, whether its reader waits, parks, or takes it. The lease lives in the stream's `Recv`, which a leased stream keeps, with its stream ID, until the application stops it, even past the FIN. When the peer has spent the connection window and no reader can make progress, the stream ranked lowest by `set_priority` (the newest on a tie) is evicted: its bytes and credit are freed, the stream is stopped without `STOP_SENDING`, and its next lease fails with `recv::Error::Evicted`; the application's `stop` then tells the peer and frees the stream once its final size is known. Leased streams that may hold credit are kept in `StreamsState::ranks`, in eviction order, so an eviction costs a logarithm rather than a scan of every stream. `MAX_DATA` goes out for any increase once the peer is within an eighth of the window (`max_data_due`), since what leases release may never add up to the usual threshold. Streams with data their reader has not looked at hold eviction off; they are kept in `StreamsState::unlooked`, so the stuck check stays constant time however many readers wait. `ConnectionStats::credit` reports the leased and available bytes, the streams evicted, and the peak credit spent. A `recv::Lease` owns the chunks it took, so a reader holds no connection borrow while it parses; `restore` or `release` hands it back, and an eviction pass waits while one is out. A port touching `Chunks` or `add_read_credits` has to keep `held` balanced. - **Owned datagrams**: a received datagram is copied out of its packet when queued, since `datagram_receive_buffer_size` charges only its payload and a slice keeps the whole receive batch alive. quinn has the same gap; not offered upstream yet. - **Stream credit on stop**: `recv::Stream::stop` on a stream whose final size is already known (finished or reset) frees it and queues `MAX_STREAMS` at once, as reading the reset does, instead of leaving the credit until the next packet arrives. Not offered upstream yet; quinn `main` has the same gap. - **Drop resets** (`quic::driver::send::Stream`): a send stream dropped before `finish` is reset with code 0, or with the peer's `STOP_SENDING` code, instead of quinn's implicit `finish`, so a cancelled writer never ends a truncated stream with a FIN. This matches moq-uring's driver and moq-net's transport contract. `quic::web` resets after the WebTransport header with the mapped code. quinn's `stream_stopped_2` finishes its stream before dropping it. Not offered upstream: it reverses a documented quinn behaviour. - **Reset secret** (`endpoint::Config::reset_secret`): derives the stateless reset key from an operator's secret with HKDF-SHA256, and keys a `HashedConnectionIdGenerator` from it too, since a restarted endpoint drops packets for connection IDs it can't validate before it would reset them. It also lowers `min_reset_interval` to 1ms, so a restart resets thousands of connections in seconds. Both drivers take one, so a restarted server resets its predecessor's connections. Not offered upstream: it only combines quinn's `reset_key` and `HashedConnectionIdGenerator::from_key`. - **Connection-blocked writers**: a deduplicated set removes a writer on finish, reset, `STOP_SENDING`, or free, so a peer withholding `MAX_DATA` cannot retain cancelled-stream history. Terminal writes report their existing error before connection credit is checked. This lives in the shared stream engine used by QUIC and QMux. Not offered upstream yet. -- **Stopped after a local reset**: moq-tokio's driver resolves `send::Finished::stopped` with `None` once the stream is reset locally, as it already did once the reset was acknowledged and the stream freed. No stream event reports a local reset, so quinn leaves a pending `stopped` waiting, and its waker registered, until the connection closes. Not offered upstream yet; quinn 0.11 has the same gap. -- **Namespaced streams** (`send`, `recv`, `stream`): quinn's `SendStream`, `RecvStream`, and their error zoo become `send::Stream` and `recv::Stream` with one `send::Error` (streams and datagrams) and one `recv::Error` (reads and leases), adding `Blocked`, and `Closed` for an ID the stream no longer has state for. `send::Stream::write` takes `&mut impl Buf`, splitting a `Bytes` source off without a copy, in place of `write` and `write_chunks`, and `Datagrams::send` takes `impl Buf` the same way, so a chained header costs no copy. An outgoing stream's `send::End` (finished and acknowledged, stopped, or reset) stays queryable until `send::Stream::release`, so a driver needs no table of its own for the fire-once `Finished` and `Stopped` events. A `send::Offset` comes only from `offset()` and is tagged to its stream, so `reset_at` and `acked` cannot be handed one past the data written. A port touching a public stream method has to be redone against these names. +- **Stopped after a local reset**: moq-tokio's driver resolves `send::Finished::stopped` with `None` once the stream is reset locally, as it already did once the reset was acknowledged and the stream freed. A local reset wakes `send::Stream::poll_end`, where quinn reports no event for it, so quinn leaves a pending `stopped` waiting, and its waker registered, until the connection closes. Not offered upstream yet; quinn 0.11 has the same gap. +- **Namespaced streams** (`send`, `recv`, `stream`): quinn's `SendStream`, `RecvStream`, and their error zoo become `send::Stream` and `recv::Stream` with one `send::Error` (streams and datagrams) and one `recv::Error` (reads and leases), adding `Blocked`, and `Closed` for an ID the stream no longer has state for. `send::Stream::write` takes `&mut impl Buf`, splitting a `Bytes` source off without a copy, in place of `write` and `write_chunks`, and `Datagrams::send` takes `impl Buf` the same way, so a chained header costs no copy. An outgoing stream's `send::End` (finished and acknowledged, stopped, or reset) stays queryable until `send::Stream::release`, so a driver needs no table of its own for how a stream ended. A `send::Offset` comes only from `offset()` and is tagged to its stream, so `reset_at` and `acked` cannot be handed one past the data written. A port touching a public stream method has to be redone against these names. - **Connection limits** (`Limits`, `Connection::set_limits`, and the drivers' `set_limits`): one validated call changes a live connection's stream limit and windows, refusing values a peer would reject and changing nothing then. moq-tokio and moq-uring re-export the type, so both runtimes check limits the same way. Not offered upstream: it only bundles quinn's existing setters. - **Indexed send buffer** (`SendBuffer::get`): each retained write keeps its stream offset, and a STREAM frame's data comes from one binary search, instead of quinn walking every write from the oldest unacknowledged one for each piece, which costs a triangular number of visits when many small writes are in flight. Not offered upstream yet. - **No receive-state recycling** (`StreamsState::stream_recv_freed`): a completed stream's `Recv` is dropped, where quinn keeps it in `free_recv` with its chunk list at peak capacity for the next stream. Recycled state stayed allocated per peak concurrent stream for the connection's lifetime: about 550 B per stream, and about 113 KiB for one that once buffered a 2,048-packet backlog. Reusing it saved only 0.1 to 0.3% of a burst's instructions. `benches/receive_bursts.rs` measures both. A cherry-pick touching `StreamRecv`, `free_recv`, or `reinit` has nothing to apply to. Not offered upstream yet. - **Bandwidth probing** (`Connection::set_probe_target`, `connection/bandwidth_probe.rs`): an application-limited sender below its target fills the headroom BBR3 opens, in Startup and in one ProbeBW\_UP per probe wait, with duplicates of STREAM data still in flight, paced to the target, so the controller's samples measure the path rather than the application. That inflight bounds an attempt like a window once it sends its first duplicate; media alone past it, as beside another flow's standing queue, leaves the sender application-limited. `Controller::probes` and `Metrics::probe_inflight` tell the sender when and how far it may fill; CUBIC never does. A duplicate is never retransmitted, a reset or stopped stream's data is never resent, and a loss retransmits only the ranges no copy delivered (`SendBuffer::retransmit`). An attempt that raises nothing for two round trips, or ends short of the target, sits out 3 s and then 4 s before retrying the same target; reaching the target or a new one retries at once. `PathStats::sent_bandwidth_probe_bytes` counts the redundant bytes. A cherry-pick touching `poll_transmit`'s space loop or `populate_packet` has to keep the probe after the ordinary frames. Not offered upstream. - **Exact datagram overhead** (`Datagram::max_payload`): `Datagrams::max_size` subtracts the frame type and the length's actual varint size from the path budget and the peer's `max_datagram_frame_size`, instead of `Datagram::SIZE_BOUND`'s worst-case 8-byte length. RFC 9221 section 3 counts the frame type and length in the limit, so a peer advertising 1200 bytes (quicly does) accepts a 1197-byte payload, where reserving 8 bytes allows only 1191. quinn has the same behaviour; not offered upstream yet. A limit too small for even an empty frame, including RFC 9221's 0 for no support, makes `max_size` return `None` instead of accepting an empty payload. -- **Waits** (`connection/waits.rs`): every wait is a `poll_*` taking the caller's `&kio::Waiter`, so a driver keeps no waiter tables of its own: `recv::Chunks::poll_next`, `recv::Stream::poll_lease`, `send::Stream::poll_write`, `poll_end`, and `poll_acked`, `Streams::poll_accept` and `poll_open`, `Datagrams::poll_recv`, and `Connection::poll_connected`, `poll_handshake_confirmed`, and `poll_closed`, over QUIC and QMux alike. The stream engine wakes each list where it reports the matching `StreamEvent` (`StreamsState::notify`), and wakes connection-blocked writers as soon as the window has room (`StreamsState::unblock`) rather than when `poll` next runs, so a port touching an event site has to keep the wake. A lost connection wakes every waiter, and each wait drains what arrived (accepted streams, readable bytes, datagrams) before it fails, a stream wait with `ConnectionLost`. Not offered upstream: quinn keeps its wakers in the async layer. +- **Waits** (`connection/waits.rs`): every wait is a `poll_*` taking the caller's `&kio::Waiter`, so a driver keeps no waiter tables of its own: `recv::Chunks::poll_next`, `recv::Stream::poll_lease` and `poll_received_reset`, `send::Stream::poll_write`, `poll_end`, and `poll_acked`, `Streams::poll_accept` and `poll_open`, `Datagrams::poll_recv`, and `Connection::poll_handshake_data`, `poll_connected`, `poll_handshake_confirmed`, and `poll_closed`, over QUIC and QMux alike, with `Connection::error` for why it was lost. quinn's `StreamEvent` and the `Event` variants a handle waited on are gone (`Event` keeps `PathUpdated`, and QMux has no events): the stream engine wakes each list where quinn queued the matching event, and wakes connection-blocked writers as soon as the window has room (`StreamsState::unblock`), so a port touching one of quinn's event sites has to wake the matching list instead. A lost connection wakes every waiter, and each wait drains what arrived (accepted streams, readable bytes, datagrams) before it fails, a stream wait with `ConnectionLost`. Not offered upstream: quinn keeps its wakers in the async layer. Recheck the BBR3 changes when porting quinn#2481 updates, or if it merges. diff --git a/rs/moq-quic/benches/common/mod.rs b/rs/moq-quic/benches/common/mod.rs index c7f6f1ad8a8..3d0bcb793c1 100644 --- a/rs/moq-quic/benches/common/mod.rs +++ b/rs/moq-quic/benches/common/mod.rs @@ -12,6 +12,7 @@ use std::{ net::{Ipv6Addr, SocketAddr}, num::NonZeroU64, sync::{Arc, Mutex}, + task::{Poll, Waker}, time::Duration, }; @@ -24,12 +25,12 @@ use criterion::{ use kio::clock::Instant; use moq_congestion::{Ack, Ce, Controller, Factory, Init, Loss, Metrics, Packet, Sent}; use moq_quic::{ - ConnectionId, ConnectionIdGenerator, Event, RandomConnectionIdGenerator, StreamEvent, VarInt, client, + ConnectionId, ConnectionIdGenerator, RandomConnectionIdGenerator, VarInt, client, connection::Connection, endpoint::{self, ConnectionHandle, DatagramEvent, Dial, Endpoint, Received as Inbound}, fuzz::Plaintext, - recv, server, - stream::Id as StreamId, + recv, send, server, + stream::{Dir, Id as StreamId}, transport, }; @@ -102,6 +103,79 @@ impl Default for Setup { } } +/// What one of the application's waiters waits on. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +enum Task { + /// Data, a FIN, or a reset to read on a stream the peer opened. + Read(ConnectionHandle, StreamId), + /// The end of a stream this side opened, to answer a STOP_SENDING. + End(ConnectionHandle, StreamId), +} + +impl Task { + fn conn(self) -> ConnectionHandle { + match self { + Self::Read(ch, _) | Self::End(ch, _) => ch, + } + } +} + +/// Queues its task when woken, as an executor's run queue would. +struct Wake { + task: Task, + ready: Arc>>, +} + +impl std::task::Wake for Wake { + fn wake(self: Arc) { + self.wake_by_ref(); + } + + fn wake_by_ref(self: &Arc) { + self.ready.lock().unwrap().push(self.task); + } +} + +/// The application's parked waiters, so it only runs what the connection woke. +#[derive(Default)] +struct Tasks { + waiters: HashMap, + /// Tasks woken, or newly started, since the application last ran them. + ready: Arc>>, +} + +impl Tasks { + /// The waiter for `task`, made on first use. + fn waiter(&mut self, task: Task) -> &kio::Waiter { + let ready = &self.ready; + self.waiters.entry(task).or_insert_with(|| { + let wake = Wake { + task, + ready: ready.clone(), + }; + kio::Waiter::new(Waker::from(Arc::new(wake))) + }) + } + + /// Runs `task` on the application's next pass. + fn start(&mut self, task: Task) { + self.ready.lock().unwrap().push(task); + } + + /// Forgets `task`, which will not wait again. + fn done(&mut self, task: Task) { + self.waiters.remove(&task); + } + + /// Takes the woken tasks of connection `ch`, leaving the others queued. + fn woken(&mut self, ch: ConnectionHandle) -> Vec { + let mut ready = self.ready.lock().unwrap(); + let (ours, rest) = ready.drain(..).partition(|task| task.conn() == ch); + *ready = rest; + ours + } +} + /// One endpoint and its connections, with an application that drains whatever arrives. pub(crate) struct Peer { pub(crate) endpoint: Endpoint, @@ -111,6 +185,7 @@ pub(crate) struct Peer { pub(crate) received: Received, /// Wall time spent in [`Peer::process`], to time one side of an exchange. pub(crate) busy: Duration, + tasks: Tasks, inbound: VecDeque<(SocketAddr, BytesMut)>, outbound: Vec<(SocketAddr, BytesMut)>, buf: Vec, @@ -125,6 +200,7 @@ impl Peer { sink: Sink::Read, received: Received::default(), busy: Duration::ZERO, + tasks: Tasks::default(), inbound: VecDeque::new(), outbound: Vec::new(), buf: Vec::new(), @@ -177,6 +253,9 @@ impl Peer { } } + // The bench loop is the driver: it polls every connection each round anyway, so its + // transmit wait never needs to wake. + let driver = kio::Waiter::noop(); for (&ch, conn) in self.conns.iter_mut() { if conn.poll_timeout().is_some_and(|at| at <= now) { conn.handle_timeout(now); @@ -186,30 +265,31 @@ impl Peer { conn.handle_event(event); } } - while let Some(event) = conn.poll() { - match event { - Event::ConnectionLost { reason } => panic!("connection lost: {reason}"), - Event::Stream(StreamEvent::Opened { .. }) if self.sink == Sink::Ignore => {} - Event::Stream(StreamEvent::Opened { dir }) => { - while let Some(id) = conn.streams().accept(dir) { - drain(conn, id, self.sink, &mut self.received); - } - } - Event::Stream(StreamEvent::Readable { id }) => drain(conn, id, self.sink, &mut self.received), - // The application answers STOP_SENDING the way a writer does: it gives up. - Event::Stream(StreamEvent::Stopped { id, error_code }) => { - let _ = conn.send_stream(id).reset(error_code); - } - Event::DatagramReceived => { - while conn.datagrams().recv().is_some() { - self.received.datagrams += 1; - } + if let Some(reason) = conn.error() { + panic!("connection lost: {reason}"); + } + if self.sink != Sink::Ignore { + for dir in [Dir::Uni, Dir::Bi] { + while let Some(id) = conn.streams().accept(dir) { + self.tasks.start(Task::Read(ch, id)); } - _ => {} + } + } + while conn.datagrams().recv().is_some() { + self.received.datagrams += 1; + } + // Runs what woke, then what those runs woke in turn. + loop { + let woken = self.tasks.woken(ch); + if woken.is_empty() { + break; + } + for task in woken { + run(conn, task, self.sink, &mut self.received, &mut self.tasks); } } buf.clear(); - while let Some(transmit) = conn.poll_transmit(now, MAX_DATAGRAMS, &mut buf) { + while let Poll::Ready(transmit) = conn.poll_transmit(&driver, now, MAX_DATAGRAMS, &mut buf) { split(&mut self.outbound, &transmit, &buf); buf.clear(); } @@ -224,65 +304,88 @@ impl Peer { } } -/// Reads or stops one stream, recording what the application saw. -fn drain(conn: &mut Connection, id: StreamId, sink: Sink, received: &mut Received) { +/// Runs one woken task, forgetting it once it will not wait again. +fn run(conn: &mut Connection, task: Task, sink: Sink, received: &mut Received, tasks: &mut Tasks) { + let done = match task { + Task::Read(_, id) => drain(conn, id, sink, received, tasks.waiter(task)), + // The application answers STOP_SENDING the way a writer does: it gives up. + Task::End(_, id) => match conn.send_stream(id).poll_end(tasks.waiter(task)) { + Poll::Pending => false, + Poll::Ready(Ok(send::End::Stopped(code))) => { + let _ = conn.send_stream(id).reset(code); + true + } + Poll::Ready(_) => true, + }, + }; + if done { + tasks.done(task); + } +} + +/// Reads or stops one stream, recording what the application saw, and returns whether it is done +/// with the stream rather than parked on it with `waiter`. +fn drain(conn: &mut Connection, id: StreamId, sink: Sink, received: &mut Received, waiter: &kio::Waiter) -> bool { if let Sink::Lease(want) = sink { - return lease(conn, id, want, received); + return lease(conn, id, want, received, waiter); } if sink == Sink::Ignore { - return; + return false; } let mut stream = conn.recv_stream(id); if sink == Sink::Stop { if stream.stop(VarInt::from_u32(0)).is_ok() { received.stopped += 1; } - return; + return true; } // A stream already stopped, finished, or reset is gone. let Ok(mut chunks) = stream.read() else { - return; + return true; }; - loop { - match chunks.next(usize::MAX) { - Ok(Some(chunk)) => received.bytes += chunk.bytes.len() as u64, - Ok(None) => { + let done = loop { + match chunks.poll_next(waiter, usize::MAX) { + Poll::Ready(Ok(Some(chunk))) => received.bytes += chunk.bytes.len() as u64, + Poll::Ready(Ok(None)) => { received.finished += 1; - break; + break true; } - Err(recv::Error::Blocked) => break, - Err(recv::Error::Reset(_)) => { + Poll::Pending => break false, + Poll::Ready(Err(recv::Error::Reset(_))) => { received.reset += 1; - break; + break true; } #[allow( unreachable_patterns, reason = "unreachable today, but keeps the suite compiling against a rewrite's new variants" )] - Err(err) => panic!("read failed: {err}"), + Poll::Ready(Err(err)) => panic!("read failed: {err}"), } - } + }; let _ = chunks.finalize(); + done } -/// Leases one stream until it waits, releasing whatever it leased. +/// Leases one stream until it waits, releasing whatever it leased, and returns whether it is done +/// with the stream rather than parked on it with `waiter`. /// /// A stream that ends is stopped, as a driver does when its reader drops, which is what /// discards its lease. -fn lease(conn: &mut Connection, id: StreamId, want: usize, received: &mut Received) { +fn lease(conn: &mut Connection, id: StreamId, want: usize, received: &mut Received, waiter: &kio::Waiter) -> bool { loop { let mut stream = conn.recv_stream(id); - let mut lease = match stream.lease(want).0 { - Ok(lease) => lease, - Err(recv::Error::Blocked | recv::Error::Closed) => return, - Err(err) => { + let mut lease = match stream.poll_lease(waiter, want).0 { + Poll::Ready(Ok(lease)) => lease, + Poll::Pending => return false, + Poll::Ready(Err(recv::Error::Closed)) => return true, + Poll::Ready(Err(err)) => { match err { recv::Error::Reset(_) => received.reset += 1, recv::Error::Evicted => received.evicted += 1, err => panic!("lease failed: {err}"), } let _ = stream.stop(VarInt::from_u32(0)); - return; + return true; } }; let fin = lease.is_fin(); @@ -295,7 +398,7 @@ fn lease(conn: &mut Connection, id: StreamId, want: usize, received: &mut Receiv if fin { received.finished += 1; let _ = stream.stop(VarInt::from_u32(0)); - return; + return true; } } } @@ -401,6 +504,18 @@ impl Pair { self.clients[0].conn() } + /// Opens a stream on the only client and writes all of `data` to it, with the client's + /// application watching how the stream ends, so it answers a STOP_SENDING. + pub(crate) fn open_uni(&mut self, data: &Bytes) -> StreamId { + assert_eq!(self.clients.len(), 1, "pair has more than one client"); + let client = &mut self.clients[0]; + let (&ch, conn) = client.conns.iter_mut().next().expect("client has a connection"); + let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); + assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), data.len()); + client.tasks.start(Task::End(ch, id)); + id + } + /// Runs every peer once and delivers what they sent, returning whether anything moved. pub(crate) fn step(&mut self) -> bool { let mut moved = false; @@ -604,9 +719,10 @@ pub(crate) fn flush(pair: &mut Pair, empty: usize) -> Vec { let mut out = Vec::new(); let server = &mut pair.server; let conn = server.conns.values_mut().next().unwrap(); + let driver = kio::Waiter::noop(); loop { server.buf.clear(); - while let Some(transmit) = conn.poll_transmit(pair.now, MAX_DATAGRAMS, &mut server.buf) { + while let Poll::Ready(transmit) = conn.poll_transmit(&driver, pair.now, MAX_DATAGRAMS, &mut server.buf) { split(&mut out, &transmit, &server.buf); server.buf.clear(); } diff --git a/rs/moq-quic/benches/receive.rs b/rs/moq-quic/benches/receive.rs index 8785e651bca..094d6d5910e 100644 --- a/rs/moq-quic/benches/receive.rs +++ b/rs/moq-quic/benches/receive.rs @@ -123,12 +123,9 @@ fn bench_lease(c: &mut Criterion) { for _ in 0..iters { pair.server.busy = Duration::ZERO; let target = pair.server.received.finished + streams as u64; - let conn = pair.client(); for _ in 0..streams { - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - let mut stream = conn.send_stream(id); - assert_eq!(stream.write(&mut data.clone()).unwrap(), data.len()); - stream.finish().unwrap(); + let id = pair.open_uni(&data); + pair.client().send_stream(id).finish().unwrap(); } pair.drive_until(|pair| pair.server.received.finished >= target); pair.settle(); @@ -171,14 +168,7 @@ fn bench_evict(c: &mut Criterion) { for _ in 0..iters { pair.server.busy = Duration::ZERO; let target = pair.server.received.evicted + 1; - let conn = pair.client(); - let ids: Vec<_> = (0..streams) - .map(|_| { - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), PACKET); - id - }) - .collect(); + let ids: Vec<_> = (0..streams).map(|_| pair.open_uni(&data)).collect(); pair.drive_until(|pair| pair.server.received.evicted >= target); busy += pair.server.busy; @@ -215,10 +205,8 @@ fn bench_evict_churn(c: &mut Criterion) { }); pair.server.sink = Sink::Lease(window); let data = payload(PACKET); - let conn = pair.client(); for _ in 0..streams { - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), PACKET); + pair.open_uni(&data); } pair.drive_until(|pair| pair.server.received.evicted >= 1); pair.settle(); @@ -230,9 +218,7 @@ fn bench_evict_churn(c: &mut Criterion) { for _ in 0..iters { pair.server.busy = Duration::ZERO; let target = pair.server.received.evicted + 1; - let conn = pair.client(); - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), PACKET); + pair.open_uni(&data); pair.drive_until(|pair| pair.server.received.evicted >= target); busy += pair.server.busy; // The evicted stream's stop and reset settle untimed. @@ -272,10 +258,8 @@ fn bench_reset_debt(c: &mut Criterion) { // Each reader waits for the whole window, which no single stream fills. pair.server.sink = Sink::Lease(window); let data = payload(PACKET); - let conn = pair.client(); for _ in 0..holders { - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), PACKET); + pair.open_uni(&data); } pair.settle(); pair.server.sink = Sink::Ignore; @@ -348,10 +332,8 @@ fn bench_stop(c: &mut Criterion) { let data = payload(bytes); b.iter(|| { let target = pair.server.received.stopped + streams as u64; - let conn = pair.client(); for _ in 0..streams { - let id = conn.streams().open(Dir::Uni, Default::default()).expect("stream limit"); - assert_eq!(conn.send_stream(id).write(&mut data.clone()).unwrap(), bytes); + pair.open_uni(&data); } pair.drive_until(|pair| { pair.server.received.stopped >= target && pair.client().streams().send_streams() == 0 diff --git a/rs/moq-quic/src/connection/mod.rs b/rs/moq-quic/src/connection/mod.rs index a9c224466a6..f04957267cf 100644 --- a/rs/moq-quic/src/connection/mod.rs +++ b/rs/moq-quic/src/connection/mod.rs @@ -106,7 +106,7 @@ pub use stats::{CreditStats, FrameStats, PathStats, Stats, UdpStats}; mod streams; pub(crate) use streams::StreamsState; -pub use streams::{ShouldTransmit, StreamEvent, Streams, recv, send}; +pub use streams::{ShouldTransmit, Streams, recv, send}; mod timer; @@ -228,8 +228,6 @@ pub struct Connection { /// Authenticated 1-RTT packets that arrived before the handshake completed, to be processed /// once it has early_1rtt: Vec, - /// Why the connection was lost, if it has been - error: Option, /// Identifies Data-space packet numbers to skip. Not used in earlier spaces. packet_number_filter: PacketNumberFilter, @@ -372,7 +370,6 @@ impl Connection { timers: TimerTable::default(), authentication_failures: 0, early_1rtt: Vec::new(), - error: None, #[cfg(test)] packet_number_filter: match config.deterministic_packet_numbers { false => PacketNumberFilter::new(&mut rng), @@ -450,6 +447,15 @@ impl Connection { } } + /// Wait for the handshake's negotiated parameters, which + /// [`handshake_data`](Self::handshake_data) then returns + /// + /// Ready once they arrived, even if the connection was lost since, or with the reason the + /// connection was lost first. + pub fn poll_handshake_data(&mut self, waiter: &kio::Waiter) -> Poll> { + self.streams.waits.poll_handshake_data(waiter) + } + /// Wait for the handshake to complete /// /// Ready once it did, even if the connection was lost since, or with the reason the @@ -470,26 +476,19 @@ impl Connection { self.streams.waits.poll_closed(waiter) } - /// Returns application-facing events + /// Why the connection was lost, if it was, as [`poll_closed`](Self::poll_closed) reports it + pub fn error(&self) -> Option<&Error> { + self.streams.waits.lost() + } + + /// Returns application-facing notifications that no `poll_*` wait covers /// /// Connections should be polled for events after: /// - a call was made to `handle_event` /// - a call was made to `handle_timeout` #[must_use] pub fn poll(&mut self) -> Option { - if let Some(x) = self.events.pop_front() { - return Some(x); - } - - if let Some(event) = self.streams.poll() { - return Some(Event::Stream(event)); - } - - if let Some(err) = self.error.take() { - return Some(Event::ConnectionLost { reason: err }); - } - - None + self.events.pop_front() } /// Return endpoint-facing events @@ -538,17 +537,36 @@ impl Connection { } } - /// Returns packets to transmit + /// Returns packets to transmit, or waits for something to send /// - /// Connections should be polled for transmit after: - /// - the application performed some I/O on the connection - /// - a call was made to `handle_event` - /// - a call was made to `handle_timeout` + /// Pending registers `waiter` until the application queues something: a write, a finish, a + /// reset or stop, a refused open, returned credit, a datagram, new limits, a local address + /// change, or a close. Only the application's calls wake it. Poll again after a call to + /// `handle_event` or `handle_timeout`, and at [`Self::poll_timeout`], which covers pacing and + /// loss recovery. A new probe target waits for the next poll, since probing only resends data + /// in flight. /// /// `max_datagrams` specifies how many datagrams can be returned inside a /// single Transmit using GSO. This must be at least 1. + pub fn poll_transmit( + &mut self, + waiter: &kio::Waiter, + now: Instant, + max_datagrams: usize, + buf: &mut Vec, + ) -> Poll { + match self.transmit(now, max_datagrams, buf) { + Some(transmit) => Poll::Ready(transmit), + None => { + self.streams.waits.register_transmit(waiter); + Poll::Pending + } + } + } + + /// The packets to transmit now, if any, as [`Self::poll_transmit`] without the wait #[must_use] - pub fn poll_transmit(&mut self, now: Instant, max_datagrams: usize, buf: &mut Vec) -> Option { + pub(crate) fn transmit(&mut self, now: Instant, max_datagrams: usize, buf: &mut Vec) -> Option { assert!(max_datagrams != 0); let max_datagrams = match self.config.segmentation_offload { false => 1, @@ -1436,16 +1454,14 @@ impl Connection { /// /// This does not ensure delivery of outstanding data. It is the application's responsibility to /// call this only when all important communications have been completed, e.g. by calling - /// [`send::Stream::finish`] on outstanding streams and waiting for the corresponding - /// [`StreamEvent::Finished`] event. + /// [`send::Stream::finish`] on outstanding streams and waiting for + /// [`send::Stream::poll_end`]. /// /// If [`Streams::send_streams`] returns 0, all outstanding stream data has been /// delivered. There may still be data from the peer that has not been received. /// /// A connection not already closed reports [`Error::LocallyClosed`] with `error_code` and - /// `reason` as its [`ConnectionLost`](Event::ConnectionLost). - /// - /// [`StreamEvent::Finished`]: crate::StreamEvent::Finished + /// `reason` from [`poll_closed`](Self::poll_closed), and fails every other wait. pub fn close(&mut self, now: Instant, error_code: VarInt, reason: String) { if !self.state.is_closed() { self.lose(Error::LocallyClosed { @@ -1539,14 +1555,15 @@ impl Connection { self.update_keys(None, false); } - /// What the handshake negotiated, once the connection emits [`Event::HandshakeDataReady`] + /// What the handshake negotiated, once [`poll_handshake_data`](Self::poll_handshake_data) is ready pub fn handshake_data(&self) -> Option { self.crypto.handshake_data() } /// The certificate chain the peer presented, end entity first, if it presented one /// - /// Complete only once the handshake finishes, signaled by [`Event::Connected`]. + /// Complete only once the handshake finishes, when [`poll_connected`](Self::poll_connected) is + /// ready. pub fn peer_identity(&self) -> Option>> { self.crypto.peer_identity() } @@ -1565,9 +1582,8 @@ impl Connection { /// Whether the connection is in the process of being established /// - /// If this returns `false`, the connection may be either established or closed, signaled by the - /// emission of a [`Connected`](Event::Connected) or [`ConnectionLost`](Event::ConnectionLost) - /// event respectively. + /// If this returns `false`, the connection may be either established or closed, as + /// [`poll_connected`](Self::poll_connected) and [`poll_closed`](Self::poll_closed) report. pub fn is_handshaking(&self) -> bool { self.state.is_handshake() } @@ -1578,9 +1594,8 @@ impl Connection { /// either peer application intentionally closes it, or when either transport layer detects an /// error such as a time-out or certificate validation failure. /// - /// A [`ConnectionLost`](Event::ConnectionLost) event is emitted with details however the - /// connection closes, with [`Error::LocallyClosed`] when the local application closed it via - /// [`close()`](Self::close). + /// [`poll_closed`](Self::poll_closed) reports why however the connection closes, with + /// [`Error::LocallyClosed`] when the local application closed it via [`close()`](Self::close). pub fn is_closed(&self) -> bool { self.state.is_closed() } @@ -1738,6 +1753,8 @@ impl Connection { self.set_max_concurrent_streams(Dir::Bi, streams); self.set_max_concurrent_streams(Dir::Uni, streams); } + // MAX_DATA or MAX_STREAMS may be due, and a new rate moves the pacing deadline. + self.streams.waits.wake_transmit(); Ok(()) } @@ -2450,7 +2467,7 @@ impl Connection { while let Some(chunk) = space.crypto_stream.read(usize::MAX) { trace!("consumed {} CRYPTO bytes", chunk.bytes.len()); if self.crypto.read_handshake(&chunk.bytes)? { - self.events.push_back(Event::HandshakeDataReady); + self.streams.waits.handshake_data(); } } @@ -2952,12 +2969,10 @@ impl Connection { // Server-only self.spaces[SpaceId::Data].pending.handshake_done = true; self.discard_space(now, SpaceId::Handshake); - self.events.push_back(Event::HandshakeConfirmed); self.streams.waits.confirmed(); trace!("handshake confirmed"); } - self.events.push_back(Event::Connected); self.streams.waits.connected(); self.state = State::Established; // Swap the handshake idle timeout for the negotiated one. @@ -3308,7 +3323,6 @@ impl Connection { .datagrams .received(datagram, &self.config.datagram.receive_buffer)? { - self.events.push_back(Event::DatagramReceived); self.streams.waits.wake_datagram(); } } @@ -3344,7 +3358,6 @@ impl Connection { if self.spaces[SpaceId::Handshake].crypto.is_some() { self.discard_space(now, SpaceId::Handshake); } - self.events.push_back(Event::HandshakeConfirmed); self.streams.waits.confirmed(); trace!("handshake confirmed"); } @@ -3439,6 +3452,8 @@ impl Connection { pub fn local_address_changed(&mut self) { self.update_rem_cid(); self.ping(); + // The PING and any RETIRE_CONNECTION_ID are due now. + self.streams.waits.wake_transmit(); } /// Switch to a previously unused remote connection ID, if possible @@ -4099,10 +4114,9 @@ impl Connection { /// Record why the connection was lost, failing every wait once it drains what it can /// - /// [`Self::poll`] reports it once as [`Event::ConnectionLost`]. + /// The first reason sticks, as [`Self::error`] and [`Self::poll_closed`] report it. fn lose(&mut self, reason: Error) { - self.streams.waits.lose(reason.clone()); - self.error = Some(reason); + self.streams.waits.lose(reason); } /// Terminate the connection instantly, without sending a close packet @@ -4413,29 +4427,13 @@ mod state { } } -/// Events of interest to the application +/// Notifications of interest to the application that no `poll_*` wait covers +/// +/// Everything an application waits for (a connection, a stream, data, credit, an end, a loss) +/// is a `poll_*` taking its [`kio::Waiter`] instead. #[derive(Debug)] #[non_exhaustive] pub enum Event { - /// The connection's handshake data is ready - HandshakeDataReady, - /// The connection was successfully established - Connected, - /// The TLS handshake was confirmed - HandshakeConfirmed, - /// The connection was lost - /// - /// Emitted when the connection is closed due to an error, a timeout, the peer closing it, or - /// the local application closing it via [`Connection::close()`], which reports - /// [`Error::LocallyClosed`]. - ConnectionLost { - /// Reason that the connection was closed - reason: Error, - }, - /// Stream events - Stream(StreamEvent), - /// One or more application datagrams have been received - DatagramReceived, /// The currently active path was updated PathUpdated, } @@ -4565,7 +4563,7 @@ mod tests { } conn.close(now, 0u32.into(), String::new()); let mut buf = Vec::new(); - assert!(conn.poll_transmit(now, 1, &mut buf).is_some()); + assert!(conn.transmit(now, 1, &mut buf).is_some()); assert!(buf.len() <= 1200); assert!(!conn.close); diff --git a/rs/moq-quic/src/connection/mux/mod.rs b/rs/moq-quic/src/connection/mux/mod.rs index 32abe827393..318eb09bf03 100644 --- a/rs/moq-quic/src/connection/mux/mod.rs +++ b/rs/moq-quic/src/connection/mux/mod.rs @@ -10,7 +10,7 @@ //! Like [`crate::connection::Connection`], it performs no I/O. Feed it each record (each frame on //! draft-00) the transport delimits with [`Connection::handle_record`], write out what //! [`Connection::poll_transmit`] returns, run [`Connection::handle_timeout`] at -//! [`Connection::poll_timeout`], and drain [`Connection::poll`] for events. +//! [`Connection::poll_timeout`], and wait on its `poll_*` forms as on QUIC. use std::{collections::VecDeque, num::NonZeroU64, task::Poll}; @@ -22,7 +22,7 @@ use super::{ datagrams::DatagramState, spaces::Retransmits, stats::FrameStats, - streams::{StreamEvent, Streams, StreamsState, recv, send}, + streams::{Streams, StreamsState, recv, send}, }; use crate::{ ApplicationClose, Dir, Duration, Instant, Limits, MAX_STREAM_COUNT, Side, TransportError, VarInt, @@ -127,22 +127,6 @@ fn limit(limit: Option, default: u64) -> VarInt { VarInt::from_u64(limit.map_or(default, NonZeroU64::get)).unwrap_or(VarInt::MAX) } -/// Events from [`Connection::poll`] -#[derive(Debug)] -pub enum Event { - /// The peer's transport parameters arrived, so streams can be opened - Connected, - /// Stream activity, as on QUIC - Stream(StreamEvent), - /// One or more datagrams arrived - DatagramReceived, - /// The connection ended - ConnectionLost { - /// Why: [`ConnectionError::Reset`] means the transport ended without a close frame - reason: ConnectionError, - }, -} - /// A QMux connection over a reliable, ordered transport pub struct Connection { side: Side, @@ -170,7 +154,6 @@ pub struct Connection { close: Option, close_sent: bool, error: Option, - events: VecDeque, /// Draft-00 frames already serialized, one per transport message queued: VecDeque, /// Where records are serialized. Each one handed out is split off, so the allocation is @@ -214,7 +197,6 @@ impl Connection { close: None, close_sent: false, error: None, - events: VecDeque::new(), queued: VecDeque::new(), out: BytesMut::new(), } @@ -333,8 +315,23 @@ impl Connection { }); } - /// The next record to write, without its Size field, or the next frame on draft-00 - pub fn poll_transmit(&mut self, now: Instant) -> Option { + /// The next record to write, without its Size field, or the next frame on draft-00, or wait + /// for one + /// + /// Pending registers `waiter` until the application queues something, as on QUIC. Poll + /// again after [`Self::handle_record`] and at [`Self::poll_timeout`]. + pub fn poll_transmit(&mut self, waiter: &kio::Waiter, now: Instant) -> Poll { + match self.transmit(now) { + Some(record) => Poll::Ready(record), + None => { + self.streams.waits.register_transmit(waiter); + Poll::Pending + } + } + } + + /// The next record to write, if any, as [`Self::poll_transmit`] without the wait + pub(crate) fn transmit(&mut self, now: Instant) -> Option { if let Some(frame) = self.queued.pop_front() { return Some(frame); } @@ -372,14 +369,6 @@ impl Connection { } } - /// The next event - pub fn poll(&mut self) -> Option { - if let Some(event) = self.events.pop_front() { - return Some(event); - } - self.streams.poll().map(Event::Stream) - } - /// Open and accept streams pub fn streams(&mut self) -> Streams<'_> { Streams { @@ -448,6 +437,8 @@ impl Connection { // A lower limit can make a credit update once deemed insignificant significant. self.streams.queue_max_stream_id(&mut self.pending); } + // MAX_DATA or MAX_STREAMS may be due. + self.streams.waits.wake_transmit(); Ok(()) } @@ -793,7 +784,6 @@ impl Connection { .datagrams .received(datagram, &Some(self.config.datagram_receive_buffer_size))? { - self.events.push_back(Event::DatagramReceived); self.streams.waits.wake_datagram(); } } @@ -856,7 +846,6 @@ impl Connection { }; self.timer.established(idle, self.config.rtt_interval, now); self.peer = Some(peer); - self.events.push_back(Event::Connected); self.streams.waits.connected(); Ok(()) } @@ -869,7 +858,6 @@ impl Connection { self.error = Some(reason.clone()); self.streams.waits.lose(reason.clone()); self.conn_state = ConnectionState::Draining; - self.events.push_back(Event::ConnectionLost { reason }); } } diff --git a/rs/moq-quic/src/connection/mux/tests.rs b/rs/moq-quic/src/connection/mux/tests.rs index 490be369baa..539db83c4f0 100644 --- a/rs/moq-quic/src/connection/mux/tests.rs +++ b/rs/moq-quic/src/connection/mux/tests.rs @@ -1,7 +1,9 @@ use bytes::{BufMut, Bytes}; use super::*; -use crate::{StreamId, TransportErrorCode, coding::BufMutExt, frame::FrameType, recv, send}; +use crate::{ + StreamId, TransportErrorCode, coding::BufMutExt, connection::waits::Counter, frame::FrameType, recv, send, +}; /// Two connections joined by an in-memory transport, on a clock the test advances struct Pair { @@ -30,11 +32,11 @@ impl Pair { fn drive(&mut self) { loop { let mut idle = true; - while let Some(record) = self.client.poll_transmit(self.now) { + while let Some(record) = self.client.transmit(self.now) { self.server.handle_record(record, self.now); idle = false; } - while let Some(record) = self.server.poll_transmit(self.now) { + while let Some(record) = self.server.transmit(self.now) { self.client.handle_record(record, self.now); idle = false; } @@ -58,10 +60,6 @@ fn config(version: Version) -> Config { } } -fn events(conn: &mut Connection) -> Vec { - std::iter::from_fn(|| conn.poll()).collect() -} - /// Read every byte currently available on `id` fn read_all(conn: &mut Connection, id: StreamId) -> (Vec, bool) { let mut recv = conn.recv_stream(id); @@ -84,10 +82,7 @@ fn read_all(conn: &mut Connection, id: StreamId) -> (Vec, bool) { } fn lost(conn: &mut Connection) -> Option { - events(conn).into_iter().find_map(|event| match event { - Event::ConnectionLost { reason } => Some(reason), - _ => None, - }) + conn.error().cloned() } #[test] @@ -96,8 +91,10 @@ fn handshake() { let mut pair = Pair::new(config(version)); assert!(pair.client.is_established(), "{version:?}"); assert!(pair.server.is_established(), "{version:?}"); - assert!(matches!(events(&mut pair.client)[..], [Event::Connected])); - assert!(matches!(events(&mut pair.server)[..], [Event::Connected])); + for conn in [&mut pair.client, &mut pair.server] { + assert_eq!(conn.poll_connected(&kio::Waiter::noop()), Poll::Ready(Ok(()))); + assert!(conn.error().is_none()); + } } } @@ -113,6 +110,8 @@ fn streams_carry_data_and_fin() { pair.drive(); } pair.client.send_stream(id).finish().unwrap(); + let end = Counter::new(); + assert!(pair.client.send_stream(id).poll_end(&end).is_pending()); pair.drive(); assert_eq!(pair.server.streams().accept(Dir::Bi), Some(id)); @@ -127,10 +126,10 @@ fn streams_carry_data_and_fin() { } assert_eq!(received, payload, "{version:?}"); // Serializing the FIN acknowledged it - assert!( - events(&mut pair.client) - .iter() - .any(|e| matches!(e, Event::Stream(StreamEvent::Finished { id: done }) if *done == id)) + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client.send_stream(id).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); } } @@ -202,7 +201,7 @@ fn close_delivers_application_close() { /// A record carrying the peer's parameters, then `frame` fn after_params(version: Version, frame: &[u8]) -> Bytes { let mut peer = Connection::new(config(version), Side::Client, Instant::now()); - let mut record = peer.poll_transmit(Instant::now()).unwrap().to_vec(); + let mut record = peer.transmit(Instant::now()).unwrap().to_vec(); record.extend_from_slice(frame); record.into() } @@ -225,7 +224,7 @@ fn rejects_prohibited_frames() { let err = transport_error(&mut server); assert_eq!(err.code, TransportErrorCode::PROTOCOL_VIOLATION, "{frame:?}"); // The violation goes out as a CONNECTION_CLOSE after our parameters - let record = server.poll_transmit(now).unwrap(); + let record = server.transmit(now).unwrap(); let mut rest = record.clone(); assert!(matches!( wire::next_frame(&mut rest).unwrap(), @@ -259,7 +258,7 @@ fn draft02_requires_params_first() { for &version in Version::ALL { let mut pair = Pair::new(config(version)); let mut repeat = Connection::new(config(version), Side::Client, now); - let params = repeat.poll_transmit(now).unwrap(); + let params = repeat.transmit(now).unwrap(); pair.server.handle_record(params, pair.now); assert_eq!( transport_error(&mut pair.server).code, @@ -330,7 +329,7 @@ fn rejects_oversized_records() { let mut server = Connection::new(config(version), Side::Server, now); server.handle_oversized(); assert_eq!(transport_error(&mut server).code, code); - assert!(server.poll_transmit(now).is_some(), "CONNECTION_CLOSE"); + assert!(server.transmit(now).is_some(), "CONNECTION_CLOSE"); } } @@ -341,7 +340,7 @@ fn records_stay_under_the_stream_ceiling() { let mut pair = Pair::new(cfg); let id = pair.client.streams().open(Dir::Uni, send::Options::default()).unwrap(); pair.client.send_stream(id).write(&mut &[0; 100_000][..]).unwrap(); - while let Some(record) = pair.client.poll_transmit(pair.now) { + while let Some(record) = pair.client.transmit(pair.now) { assert!(record.len() <= MAX_RECORD_BUDGET); pair.server.handle_record(record, pair.now); } @@ -352,14 +351,14 @@ fn draft00_sends_one_length_prefixed_frame_per_message() { let now = Instant::now(); let mut client = Connection::new(config(Version::QMux00), Side::Client, now); let mut server = Connection::new(config(Version::QMux00), Side::Server, now); - while let Some(frame) = server.poll_transmit(now) { + while let Some(frame) = server.transmit(now) { client.handle_record(frame, now); } let id = client.streams().open(Dir::Uni, send::Options::default()).unwrap(); client.send_stream(id).write(&mut &[3; 40_000][..]).unwrap(); client.send_stream(id).finish().unwrap(); let mut messages = 0; - while let Some(message) = client.poll_transmit(now) { + while let Some(message) = client.transmit(now) { assert!(message.len() <= MAX_FRAME_SIZE); let ty = message[0]; if (0x08..=0x0f).contains(&ty) { @@ -472,16 +471,14 @@ fn datagrams() { let mut pair = Pair::new(config(Version::QMux02)); let max = pair.client.datagrams().max_size().unwrap(); assert!(max > 16_000); + let waiter = Counter::new(); + assert!(pair.server.datagrams().poll_recv(&waiter).is_pending()); pair.client .datagrams() .send(Bytes::from_static(b"hello"), send::Options::default()) .unwrap(); pair.drive(); - assert!( - events(&mut pair.server) - .iter() - .any(|e| matches!(e, Event::DatagramReceived)) - ); + assert_eq!(waiter.wakes(), 1); assert_eq!(pair.server.datagrams().recv().unwrap(), &b"hello"[..]); // Draft-00 has none @@ -600,8 +597,8 @@ fn idle_timeout_reclaims_a_silent_peer() { // The server never hears back: its keep-alives go out, nothing answers for _ in 0..10 { pair.advance(Duration::from_secs(5)); - while pair.server.poll_transmit(pair.now).is_some() {} - while pair.client.poll_transmit(pair.now).is_some() {} + while pair.server.transmit(pair.now).is_some() {} + while pair.client.transmit(pair.now).is_some() {} } assert!(matches!(lost(&mut pair.server), Some(ConnectionError::TimedOut))); } @@ -662,7 +659,7 @@ fn golden_transport_parameters() { ]; for (version, hex) in expected { let mut client = Connection::new(config(version), Side::Client, Instant::now()); - let record = client.poll_transmit(Instant::now()).unwrap(); + let record = client.transmit(Instant::now()).unwrap(); let actual: String = record.iter().map(|b| format!("{b:02x}")).collect(); assert_eq!(actual, hex, "{version:?}"); } @@ -676,8 +673,8 @@ fn golden_records() { for &version in Version::ALL { let mut pair = Pair::new(config(version)); let record = |conn: &mut Connection, now| { - let record = conn.poll_transmit(now).unwrap(); - assert!(conn.poll_transmit(now).is_none()); + let record = conn.transmit(now).unwrap(); + assert!(conn.transmit(now).is_none()); record.iter().map(|b| format!("{b:02x}")).collect::() }; @@ -707,7 +704,7 @@ fn golden_records() { /// The client's STREAM bytes in the order its records carry them, as `(stream, bytes)` runs fn stream_order(pair: &mut Pair) -> Vec<(u64, usize)> { let mut order: Vec<(u64, usize)> = Vec::new(); - while let Some(record) = pair.client.poll_transmit(pair.now) { + while let Some(record) = pair.client.transmit(pair.now) { let mut rest = record.clone(); while let Some(frame) = wire::next_frame(&mut rest).unwrap() { let (wire::Frame::Quic(frame::Frame::Stream(stream)) | wire::Frame::Continue(stream)) = frame else { diff --git a/rs/moq-quic/src/connection/streams/lease.rs b/rs/moq-quic/src/connection/streams/lease.rs index 6a7f64416cc..434a05b2a28 100644 --- a/rs/moq-quic/src/connection/streams/lease.rs +++ b/rs/moq-quic/src/connection/streams/lease.rs @@ -8,7 +8,7 @@ use std::task::Poll; use bytes::{Buf, BufMut, Bytes, BytesMut}; use super::recv::{Error, Stream}; -use super::{ShouldTransmit, StreamEvent, StreamId}; +use super::{ShouldTransmit, StreamId}; use super::{StreamsState, state::get_or_insert_recv}; use crate::VarInt; use crate::connection::assembler::Assembler; @@ -265,7 +265,7 @@ impl Stream<'_> { /// back before leasing again. If the returned [`ShouldTransmit::should_transmit()`] is /// `true`, transmit soon, whatever the result. /// - /// [`Error::Blocked`] records the `want`, so the next [`StreamEvent::Readable`] for this + /// [`Error::Blocked`] records the `want`, so the next wake of a [`Self::poll_lease()`] on this /// stream means it may be met, and so that a connection window every reader is waiting on /// can be recovered by evicting the stream ranked lowest by [`Self::set_priority()`]. /// @@ -288,6 +288,11 @@ impl Stream<'_> { /// - when the last lease of the stream was not handed back #[must_use = "a lease may need a transmit"] pub fn lease(&mut self, want: usize) -> (Result, ShouldTransmit) { + let (result, transmit) = self.lease_inner(want); + (result, self.transmit(transmit)) + } + + fn lease_inner(&mut self, want: usize) -> (Result, ShouldTransmit) { // Bytes out in a lease do not count: a unit that outgrows the window with them is one // the stuck window evicts by rank, so it gives way like any other rather than failing // alone. @@ -368,7 +373,8 @@ impl Stream<'_> { self.put_back(lease); // Only a stuck pass that waited reruns: a reader handing back what it looked at is not // stuck itself, and one that is waits for more with its next lease. - ShouldTransmit(self.state.stuck_waiting && self.unstick()) + let transmit = ShouldTransmit(self.state.stuck_waiting && self.unstick()); + self.transmit(transmit) } /// Hand a lease back, returning the connection credit of every byte read from it or from an @@ -378,7 +384,16 @@ impl Stream<'_> { /// of the larger window. pub fn release(&mut self, lease: Lease) -> ShouldTransmit { self.put_back(lease); - self.release_credit() + let transmit = self.release_credit(); + self.transmit(transmit) + } + + /// Wake the driver when `transmit` says a frame is due, passing it on + fn transmit(&mut self, transmit: ShouldTransmit) -> ShouldTransmit { + if transmit.should_transmit() { + self.state.waits.wake_transmit(); + } + transmit } /// Return the credit of every byte read out of a lease. @@ -425,7 +440,8 @@ impl Stream<'_> { /// Return the credit of every byte read out, as [`Self::release()`] does. #[cfg(test)] pub(crate) fn release_taken(&mut self) -> ShouldTransmit { - self.release_credit() + let transmit = self.release_credit(); + self.transmit(transmit) } /// Put the bytes `lease` did not read back in front of the stream's lease. @@ -569,7 +585,7 @@ impl StreamsState { /// arrival pays for one look at most. /// /// Returns `None` if nothing was evicted, or whether to transmit the `MAX_DATA` the evictions - /// freed; each evicted stream's reader is woken by a [`StreamEvent::Readable`]. + /// freed; each evicted stream's reader wakes. pub(super) fn evict_if_stuck(&mut self) -> Option { self.stuck_waiting = false; // The window, not the credit a rate cap paces, says whether memory is stuck: a capped @@ -661,7 +677,7 @@ impl StreamsState { self.held -= leased; self.unlooked.remove(&id); self.evicted += 1; - self.notify(StreamEvent::Readable { id }); + self.waits.wake(id, Wait::Read); self.add_read_credits(leased + unread) } diff --git a/rs/moq-quic/src/connection/streams/mod.rs b/rs/moq-quic/src/connection/streams/mod.rs index 017a27ed44c..1110ed3c09b 100644 --- a/rs/moq-quic/src/connection/streams/mod.rs +++ b/rs/moq-quic/src/connection/streams/mod.rs @@ -3,7 +3,7 @@ use std::task::Poll; use tracing::trace; use super::spaces::{Retransmits, ThinRetransmits}; -use crate::{Dir, StreamId, VarInt, connection::Error, frame}; +use crate::{Dir, StreamId, connection::Error, frame}; mod lease; pub mod recv; @@ -48,6 +48,8 @@ impl<'a> Streams<'a> { if self.state.next[dir as usize] >= self.state.max[dir as usize] { self.state.streams_blocked[dir as usize] = true; self.state.open_refused[dir as usize] = true; + // STREAMS_BLOCKED + self.state.waits.wake_transmit(); return None; } @@ -155,6 +157,7 @@ impl StreamsState { ) -> bool { let group = options.group.map(|g| g.index(self.tag)); let frame = datagram.size(); + self.waits.wake_transmit(); self.pending .push_datagram(group, options.priority, datagram, frame, budget) } @@ -180,59 +183,6 @@ impl StreamsState { } } -/// Application events about streams -#[derive(Debug, PartialEq, Eq)] -pub enum StreamEvent { - /// One or more new streams has been opened and can be accepted - /// - /// A separate [`StreamEvent::Readable`] is emitted once data arrives. - Opened { - /// Directionality for which streams have been opened - dir: Dir, - }, - /// A currently open stream likely has data or errors waiting to be read - Readable { - /// Which stream is now readable - id: StreamId, - }, - /// A formerly write-blocked stream might be ready for a write or have been stopped - /// - /// Only generated for streams that are currently open. - Writable { - /// Which stream is now writable - id: StreamId, - }, - /// A finished stream has been fully acknowledged or stopped - Finished { - /// Which stream has been finished - id: StreamId, - }, - /// The answer changed for the lowest offset a [`send::Stream::acked`] call is waiting for - /// - /// The stream's contiguous acknowledged prefix reached it, or a reset or STOP_SENDING may have - /// failed any waiter. Every waiter on the stream polls again: one at or below `offset` is - /// acknowledged, and one past it learns its error or registers again. Reported before a [`StreamEvent::Finished`] from the same ACK. - /// Not reported when the connection closes, which fails [`send::Stream::acked`] instead. - Acked { - /// Which stream was acknowledged - id: StreamId, - /// The end of the stream's contiguous acknowledged prefix - offset: u64, - }, - /// The peer asked us to stop sending on an outgoing stream - Stopped { - /// Which stream has been stopped - id: StreamId, - /// Error code supplied by the peer - error_code: VarInt, - }, - /// At least one new stream of a certain directionality may be opened - Available { - /// Directionality for which streams are newly available - dir: Dir, - }, -} - /// Indicates whether a frame needs to be transmitted /// /// This type wraps around bool and uses the `#[must_use]` attribute in order diff --git a/rs/moq-quic/src/connection/streams/recv.rs b/rs/moq-quic/src/connection/streams/recv.rs index 092f579cafc..7bcf6c1b15e 100644 --- a/rs/moq-quic/src/connection/streams/recv.rs +++ b/rs/moq-quic/src/connection/streams/recv.rs @@ -76,6 +76,8 @@ impl Stream<'_> { self.unstick(); // A reader parked on the stream learns it is closed. self.state.waits.wake(self.id, Wait::Read); + // STOP_SENDING, or the credit the stop returned. + self.state.waits.wake_transmit(); Ok(()) } @@ -114,9 +116,28 @@ impl Stream<'_> { self.state.queue_max_stream_id(self.pending); // The credit returned may only have paid a shrunk window's debt. self.unstick(); + self.state.waits.wake_transmit(); Ok(Some(code)) } + + /// [`Self::received_reset`], or wait for the peer to reset the stream + /// + /// Pending registers `waiter` until data, a FIN, or a reset arrives, after which the answer + /// may still be no reset. Fails with [`Error::Closed`] once the stream is gone, as when every + /// byte up to a FIN was read, and with [`Error::ConnectionLost`] once the connection is lost + /// without a reset. + pub fn poll_received_reset(&mut self, waiter: &kio::Waiter) -> Poll> { + match self.received_reset() { + Ok(Some(code)) => Poll::Ready(Ok(code)), + Err(err) => Poll::Ready(Err(err)), + Ok(None) if self.state.waits.lost().is_some() => Poll::Ready(Err(Error::ConnectionLost)), + Ok(None) => { + self.state.waits.register(self.id, Wait::Read, waiter); + Poll::Pending + } + } + } } #[derive(Debug, Default)] @@ -545,6 +566,9 @@ impl<'a> Chunks<'a> { let max_data = self.streams.add_read_credits(self.read); self.pending.max_data |= max_data.0; should_transmit |= max_data.0; + if should_transmit { + self.streams.waits.wake_transmit(); + } ShouldTransmit(should_transmit) } } diff --git a/rs/moq-quic/src/connection/streams/send.rs b/rs/moq-quic/src/connection/streams/send.rs index 7ced48102bc..eb815152b69 100644 --- a/rs/moq-quic/src/connection/streams/send.rs +++ b/rs/moq-quic/src/connection/streams/send.rs @@ -6,7 +6,7 @@ use bytes::Buf; use thiserror::Error; use tracing::trace; -use super::{StreamEvent, StreamsState, schedule::Slot}; +use super::{StreamsState, schedule::Slot}; use crate::connection::{ConnectionTag, State as ConnState, send_buffer::SendBuffer, spaces::Retransmits, waits::Wait}; use crate::{StreamId, VarInt, frame}; @@ -262,21 +262,20 @@ impl Send { Ok(false) } - /// The acknowledged prefix to report once it reaches the offset a waiter is waiting for - pub(super) fn take_acked(&mut self) -> Option { - let acked = self.pending.acked(); - if self.notify.is_none_or(|notify| acked < notify) { - return None; + /// Whether the acknowledged prefix reached the lowest offset a waiter is waiting for, which + /// then wakes every waiter on the stream + pub(super) fn take_acked(&mut self) -> bool { + if self.notify.is_none_or(|notify| self.pending.acked() < notify) { + return false; } self.notify = None; - Some(acked) + true } - /// The acknowledged prefix to report to every waiter after a reset or STOP_SENDING, which may - /// fail any of them - pub(super) fn take_waiting(&mut self) -> Option { - self.notify.take()?; - Some(self.pending.acked()) + /// Whether a waiter is waiting on the acknowledged prefix, which a reset or STOP_SENDING may + /// fail, so every waiter on the stream wakes + pub(super) fn take_waiting(&mut self) -> bool { + self.notify.take().is_some() } /// Handle increase to stream-level flow control limit @@ -355,8 +354,8 @@ impl<'a> Stream<'a> { /// Queue as much of `data` as the stream and connection accept now, returning how many bytes /// /// Each chunk is taken with [`Buf::copy_to_bytes`], which a [`Bytes`](bytes::Bytes) source - /// splits off without copying. Fails with [`Error::Blocked`] when nothing fits; a - /// [`StreamEvent::Writable`] says when to retry. + /// splits off without copying. Fails with [`Error::Blocked`] when nothing fits; + /// [`Self::poll_write`] waits for room instead. pub fn write(&mut self, data: &mut impl Buf) -> Result { if self.conn_state.is_closed() { trace!(%self.id, "write blocked; connection draining"); @@ -382,10 +381,12 @@ impl<'a> Stream<'a> { { self.state.data_blocked_limit = Some(self.state.max_data); self.pending.data_blocked = true; + self.state.waits.wake_transmit(); } } else if stream.data_blocked_limit != Some(stream.max_data) { stream.data_blocked_limit = Some(stream.max_data); self.pending.stream_data_blocked.insert(self.id); + self.state.waits.wake_transmit(); } return Err(Error::Blocked); } @@ -397,6 +398,7 @@ impl<'a> Stream<'a> { if !was_pending { self.state.pending.push(self.id, &mut stream.slot, stream.priority); } + self.state.waits.wake_transmit(); Ok(written) } @@ -425,7 +427,12 @@ impl<'a> Stream<'a> { Some(Some(stream)) if stream.released => Err(Error::Closed), Some(Some(stream)) => Ok(stream.end()), Some(None) => Ok(None), - None => self.state.ends.get(&self.id).map(|&end| Some(end)).ok_or(Error::Closed), + None => self + .state + .ends + .get(&self.id) + .map(|&(end, _)| Some(end)) + .ok_or(Error::Closed), } } @@ -479,6 +486,7 @@ impl<'a> Stream<'a> { } // A writer waiting on it learns it is closed. self.state.waits.wake(self.id, Wait::Write); + self.state.waits.wake_transmit(); Ok(()) } @@ -508,9 +516,8 @@ impl<'a> Stream<'a> { /// Whether the peer acknowledged every byte before `offset` /// - /// A `false` notes to report a [`StreamEvent::Acked`] once the answer changes, so poll again - /// then. The stream holds only its lowest waiting offset, so every waiter on it polls again - /// on that event. Like [`End::Finished`], an acknowledgment only means the peer's QUIC stack + /// A `false` notes to wake the stream's [`Self::poll_acked`] waiters once the answer changes. + /// The stream holds only its lowest waiting offset, so every waiter on it polls again then. Like [`End::Finished`], an acknowledgment only means the peer's QUIC stack /// holds the data, not that its application read it. /// /// Fails with [`Error::Closed`] once the peer will never acknowledge it, or no longer reads @@ -529,8 +536,10 @@ impl<'a> Stream<'a> { // Nothing was written, so an offset of it is 0. Some(None) => Ok(true), Some(Some(_)) => Err(Error::Closed), + // A reset keeps answering for the prefix the peer acknowledged before it freed the stream. None => match self.state.ends.get(&self.id) { - Some(End::Finished) => Ok(true), + Some((End::Finished, _)) => Ok(true), + Some((End::Reset(_), acked)) if offset <= *acked => Ok(true), _ => Err(Error::Closed), }, } @@ -613,9 +622,10 @@ impl<'a> Stream<'a> { } if let Some(reliable_size) = send { self.pending.reset_stream.push((self.id, reliable_size)); + self.state.waits.wake_transmit(); } - if let Some(offset) = waiting { - self.state.notify(StreamEvent::Acked { id: self.id, offset }); + if waiting { + self.state.waits.wake(self.id, Wait::Acked); } // The stream ended, and a writer waiting on it learns it is closed. self.state.waits.wake(self.id, Wait::End); diff --git a/rs/moq-quic/src/connection/streams/state.rs b/rs/moq-quic/src/connection/streams/state.rs index 37eff831a67..1f147533e1f 100644 --- a/rs/moq-quic/src/connection/streams/state.rs +++ b/rs/moq-quic/src/connection/streams/state.rs @@ -1,5 +1,5 @@ use std::{ - collections::{BTreeSet, VecDeque, hash_map}, + collections::{BTreeSet, hash_map}, convert::TryFrom, mem, ops::Range, @@ -11,7 +11,7 @@ use rustc_hash::{FxBuildHasher, FxHashMap, FxHashSet}; use tracing::{debug, trace}; use super::{ - Next, Recv, Retransmits, Schedule, Send, SendState, ShouldTransmit, StreamEvent, StreamHalf, ThinRetransmits, + Next, Recv, Retransmits, Schedule, Send, SendState, ShouldTransmit, StreamHalf, ThinRetransmits, lease::Rank, send::{End, Error}, }; @@ -39,8 +39,9 @@ pub(crate) struct StreamsState { pub(super) side: Side, // Set of streams that are currently open, or could be immediately opened by the peer pub(super) send: FxHashMap>>, - /// How each send stream whose state is gone ended, until the application releases its ID - pub(super) ends: FxHashMap, + /// How each send stream whose state is gone ended, and the end of its acknowledged prefix, + /// until the application releases its ID + pub(super) ends: FxHashMap, /// Receive state by stream, `None` until a frame or the application first touches it pub(super) recv: FxHashMap>>, pub(super) next: [u64; 2], @@ -59,9 +60,6 @@ pub(crate) struct StreamsState { max_concurrent_remote_count: [u64; 2], /// Lowest remotely-initiated stream index that haven't actually been opened by the peer pub(super) next_remote: [u64; 2], - /// Whether the remote endpoint has opened any streams the application doesn't know about yet, - /// per directionality - opened: [bool; 2], // Next to report to the application, once opened pub(super) next_reported_remote: [u64; 2], /// Number of outbound streams @@ -72,8 +70,7 @@ pub(crate) struct StreamsState { /// Streams with outgoing data queued and outgoing datagrams, in the order to send them pub(super) pending: Schedule, - pub(super) events: VecDeque, - /// The waiters on the streams and the connection, woken alongside the events + /// The waiters on the streams and the connection, woken as their state changes pub(crate) waits: Waits, /// Streams blocked on connection-level flow control or stream window space /// @@ -177,11 +174,9 @@ impl StreamsState { allocated_remote_count: [max_remote_bi.into(), max_remote_uni.into()], max_concurrent_remote_count: [max_remote_bi.into(), max_remote_uni.into()], next_remote: [0, 0], - opened: [false, false], next_reported_remote: [0, 0], send_streams: 0, pending: Schedule::new(), - events: VecDeque::new(), waits: Waits::default(), connection_blocked: IndexSet::default(), max_data: 0, @@ -230,7 +225,7 @@ impl StreamsState { let count = count.into(); // A caller can attempt an open before the peer's limit is known. if count > self.max[dir as usize] && mem::take(&mut self.open_refused[dir as usize]) { - self.notify(StreamEvent::Available { dir }); + self.waits.wake_open(dir); } self.max[dir as usize] = count; } @@ -281,7 +276,7 @@ impl StreamsState { self.spend(new_bytes); let transmit = if !stopped { - self.notify(StreamEvent::Readable { id }); + self.waits.wake(id, Wait::Read); self.ranks.extend(rank); self.unlooked.insert(id); // Credit released while the peer still had room may not have been worth a frame @@ -384,7 +379,7 @@ impl StreamsState { } new_bytes } else { - self.notify(StreamEvent::Readable { id }); + self.waits.wake(id, Wait::Read); self.ranks.extend(rank); self.unlooked.insert(id); freed @@ -432,10 +427,11 @@ impl StreamsState { self.pending.remove(&mut stream.slot); } if stopped { - self.notify(StreamEvent::Stopped { id, error_code }); + self.waits.wake(id, Wait::Write); + self.waits.wake(id, Wait::End); } - if let Some(offset) = waiting { - self.notify(StreamEvent::Acked { id, offset }); + if waiting { + self.waits.wake(id, Wait::Acked); } self.unblock(); Ok(reset) @@ -837,8 +833,8 @@ impl StreamsState { None => entry.remove(), Some(_) => None, }; - if let Some(offset) = acked { - self.notify(StreamEvent::Acked { id, offset }); + if acked { + self.waits.wake(id, Wait::Acked); } self.unblock(); if let Some(held) = held { @@ -850,7 +846,7 @@ impl StreamsState { } self.stream_freed(id, StreamHalf::Send); if !reset { - self.notify(StreamEvent::Finished { id }); + self.waits.wake(id, Wait::End); } None } @@ -879,7 +875,7 @@ impl StreamsState { *current = count; self.streams_blocked[dir as usize] = false; self.open_refused[dir as usize] = false; - self.notify(StreamEvent::Available { dir }); + self.waits.wake_open(dir); } Ok(()) @@ -916,7 +912,7 @@ impl StreamsState { return; } if self.write_limit() > 0 { - self.notify(StreamEvent::Writable { id }); + self.waits.wake(id, Wait::Write); return; } // Still blocked on the connection window, so wait in the list that wakes when it relaxes @@ -935,32 +931,6 @@ impl StreamsState { .min(self.send_window.saturating_sub(self.buffered_data)) } - /// Yield stream events - pub(crate) fn poll(&mut self) -> Option { - if let Some(dir) = Dir::iter().find(|&i| mem::replace(&mut self.opened[i as usize], false)) { - return Some(StreamEvent::Opened { dir }); - } - - self.events.pop_front() - } - - /// Report `event`, and wake the waiters it concerns - pub(super) fn notify(&mut self, event: StreamEvent) { - match event { - StreamEvent::Opened { dir } => self.waits.wake_accept(dir), - StreamEvent::Readable { id } => self.waits.wake(id, Wait::Read), - StreamEvent::Writable { id } => self.waits.wake(id, Wait::Write), - StreamEvent::Finished { id } => self.waits.wake(id, Wait::End), - StreamEvent::Acked { id, .. } => self.waits.wake(id, Wait::Acked), - StreamEvent::Stopped { id, .. } => { - self.waits.wake(id, Wait::Write); - self.waits.wake(id, Wait::End); - } - StreamEvent::Available { dir } => self.waits.wake_open(dir), - } - self.events.push_back(event); - } - /// Wake the writers blocked on the connection window, once it has room /// /// Called wherever the room grows: the peer raising `MAX_DATA`, or buffered data leaving the @@ -977,7 +947,7 @@ impl StreamsState { // If it's no longer sensible to write to a stream (even to detect an error) then don't // report it. if stream.is_writable() && stream.stop_reason.is_none() && stream.max_data > stream.offset() { - self.notify(StreamEvent::Writable { id }); + self.waits.wake(id, Wait::Write); } } } @@ -1129,7 +1099,6 @@ impl StreamsState { } } self.next_remote[dir_idx] = id.index() + 1; - self.opened[dir_idx] = true; self.waits.wake_accept(id.dir()); } @@ -1277,7 +1246,7 @@ impl StreamsState { if !stream.released && let Some(end) = stream.end() { - self.ends.insert(id, end); + self.ends.insert(id, (end, stream.pending.acked())); } } @@ -1365,8 +1334,10 @@ mod tests { connection::State as ConnState, connection::Streams, connection::streams::{recv, send}, + connection::waits::Counter, }; use bytes::{Buf, Bytes}; + use std::task::Poll; fn make(side: Side) -> StreamsState { StreamsState::new( @@ -1492,10 +1463,13 @@ mod tests { pending: &mut pending, conn_state: &state, }; - assert_eq!(stream.write(&mut &b"blocked"[..]), Err(send::Error::Blocked)); + let (write, end) = (Counter::new(), Counter::new()); + assert!(stream.poll_write(&write, &mut &b"blocked"[..]).is_pending()); + assert!(stream.poll_end(&end).is_pending()); let code = 7u32.into(); stream.state.received_stop_sending(id, code).unwrap(); - assert_eq!(stream.state.poll(), Some(StreamEvent::Stopped { id, error_code: code })); + assert_eq!((write.wakes(), end.wakes()), (1, 1)); + assert_eq!(stream.poll_end(&end), Poll::Ready(Ok(send::End::Stopped(code)))); assert_eq!(stream.write(&mut &b"stopped"[..]), Err(send::Error::Stopped(code))); assert_eq!(stream.write(&mut &[][..]), Err(send::Error::Stopped(code))); let mut chunk = Bytes::from_static(b"stopped"); @@ -1516,8 +1490,10 @@ mod tests { ..TransportParameters::default() }); let (mut pending, state) = (Retransmits::default(), ConnState::Established); - let ids: Vec<_> = (0..4) - .map(|_| { + let writers: Vec<_> = (0..4).map(|_| Counter::new()).collect(); + let ids: Vec<_> = writers + .iter() + .map(|writer| { let id = Streams { state: &mut server, conn_state: &state, @@ -1531,7 +1507,7 @@ mod tests { conn_state: &state, }; assert_eq!(stream.write(&mut &b"x"[..]), Err(send::Error::Blocked)); - assert_eq!(stream.write(&mut &b"x"[..]), Err(send::Error::Blocked)); + assert!(stream.poll_write(writer, &mut &b"x"[..]).is_pending()); id }) .collect(); @@ -1556,12 +1532,15 @@ mod tests { for &id in &ids[..3] { server.received_max_stream_data(id, 2048).unwrap(); } - server.events.clear(); + let wakes = |writers: &[Counter]| writers.iter().map(Counter::wakes).collect::>(); + let before = wakes(&writers); + assert_eq!(before[3], 0); assert_eq!(server.connection_blocked.len(), 1); server.received_max_data(1024u32.into()); - assert_eq!(server.poll(), Some(StreamEvent::Writable { id: ids[3] })); + let mut expected = before; + expected[3] = 1; + assert_eq!(wakes(&writers), expected, "only the live writer wakes"); assert!(server.connection_blocked.is_empty()); - assert_eq!(server.poll(), None); } #[test] @@ -2075,18 +2054,23 @@ mod tests { }; let error_code = 0u32.into(); + let end = Counter::new(); + assert!(stream.poll_end(&end).is_pending()); stream.state.received_stop_sending(id, error_code).unwrap(); - assert!(stream.state.events.contains(&StreamEvent::Stopped { id, error_code })); - stream.state.events.clear(); + assert_eq!(end.wakes(), 1); + assert_eq!(stream.poll_end(&end), Poll::Ready(Ok(send::End::Stopped(error_code)))); assert_eq!(stream.write(&mut &[][..]), Err(send::Error::Stopped(error_code))); stream.reset(0u32.into()).unwrap(); assert_eq!(stream.write(&mut &[][..]), Err(send::Error::Closed)); - // A duplicate frame is a no-op + // A duplicate frame is a no-op, so it wakes nobody parked on the stream + let (write, end) = (Counter::new(), Counter::new()); + stream.state.waits.register(id, Wait::Write, &write); + stream.state.waits.register(id, Wait::End, &end); stream.state.received_stop_sending(id, error_code).unwrap(); - assert!(stream.state.events.is_empty()); + assert_eq!((write.wakes(), end.wakes()), (0, 0)); } #[test] @@ -2967,6 +2951,8 @@ mod tests { let larger_send_window = initial_send_window * 2; // Set `initial_max_data` larger than `send_window` so we're limited by local flow control + let open = Counter::new(); + server.waits.register_open(Dir::Uni, &open); server.set_params(&TransportParameters { initial_max_data: VarInt::MAX, initial_max_stream_data_uni: VarInt::MAX, @@ -2975,7 +2961,7 @@ mod tests { }); assert_eq!(server.write_limit(), initial_send_window); - assert_eq!(server.poll(), None); + assert_eq!(open.wakes(), 0, "no open was refused"); let mut retransmits = Retransmits::default(); let conn_state = ConnState::Established; @@ -3001,17 +2987,18 @@ mod tests { assert_eq!(stream.write(&mut &data[..]), Ok(initial_send_len)); // Try to write the same data again, observe that it's blocked - assert_eq!(stream.write(&mut &data[..]), Err(send::Error::Blocked)); + let writer = Counter::new(); + assert!(stream.poll_write(&writer, &mut &data[..]).is_pending()); - // Check that we get a `Writable` event after increasing the send window + // Check that the writer wakes after increasing the send window stream.state.set_send_window(larger_send_window); - assert_eq!(stream.state.poll(), Some(StreamEvent::Writable { id: stream_id })); + assert_eq!(writer.wakes(), 1); // Check that the stream accepts the exact same amount of data again assert_eq!(stream.write(&mut &data[..]), Ok(initial_send_len)); - assert_eq!(stream.write(&mut &data[..]), Err(send::Error::Blocked)); + assert!(stream.poll_write(&writer, &mut &data[..]).is_pending()); - assert_eq!(stream.state.poll(), None); + assert_eq!(writer.wakes(), 1); // Ack the data stream.state.received_ack_of(frame::StreamMeta { @@ -3020,7 +3007,7 @@ mod tests { fin: false, }); - assert_eq!(stream.state.poll(), Some(StreamEvent::Writable { id: stream_id })); + assert_eq!(writer.wakes(), 2); // Check that our full send window is available again assert_eq!(stream.write(&mut &data[..]), Ok(initial_send_len)); @@ -3036,6 +3023,8 @@ mod tests { let smaller_send_window = server.send_window / 2; // Set `initial_max_data` larger than `send_window` so we're limited by local flow control + let open = Counter::new(); + server.waits.register_open(Dir::Uni, &open); server.set_params(&TransportParameters { initial_max_data: VarInt::MAX, initial_max_stream_data_uni: VarInt::MAX, @@ -3044,7 +3033,7 @@ mod tests { }); assert_eq!(server.write_limit(), initial_send_window); - assert_eq!(server.poll(), None); + assert_eq!(open.wakes(), 0, "no open was refused"); let mut retransmits = Retransmits::default(); let conn_state = ConnState::Established; @@ -3069,15 +3058,16 @@ mod tests { // Assert that the full send window is accepted assert_eq!(stream.write(&mut &data[..]), Ok(initial_send_len)); - assert_eq!(stream.write(&mut &data[..]), Err(send::Error::Blocked)); + let writer = Counter::new(); + assert!(stream.poll_write(&writer, &mut &data[..]).is_pending()); assert_eq!(stream.state.write_limit(), 0); - assert_eq!(stream.state.poll(), None); + assert_eq!(writer.wakes(), 0); // Shrink our send window, assert that it's still not writable stream.state.set_send_window(smaller_send_window); assert_eq!(stream.state.write_limit(), 0); - assert_eq!(stream.state.poll(), None); + assert_eq!(writer.wakes(), 0); // Assert that data is still not accepted assert_eq!(stream.write(&mut &data[..]), Err(send::Error::Blocked)); @@ -3098,8 +3088,8 @@ mod tests { fin: false, }); - // This should generate a `Writable` event - assert_eq!(stream.state.poll(), Some(StreamEvent::Writable { id: stream_id })); + // This should wake the writer + assert_eq!(writer.wakes(), 1); assert_eq!(stream.state.write_limit(), smaller_send_window); // Assert that only `smaller_send_window` bytes are accepted @@ -3476,6 +3466,9 @@ mod tests { let sent = stream.state.write_frames(&mut buf, 0..100, true).streams; assert_eq!(sent[0].offsets, 0..11); stream.reset_at(stream.offset().rewind(4), 7u32.into()).unwrap(); + // Nobody was refused an open, so the stream credit freed below wakes nobody + let open = Counter::new(); + server.waits.register_open(Dir::Uni, &open); assert_eq!(server.buffered_data, 4); // Only the lost prefix is retransmitted, without the FIN of a stream that was never finished @@ -3493,7 +3486,7 @@ mod tests { server.received_ack_of(resent[0].clone()); assert_eq!(server.send_streams, 0); assert_eq!(server.buffered_data, 0); - assert_eq!(server.poll(), None); + assert_eq!(open.wakes(), 0); } #[test] @@ -3725,15 +3718,45 @@ mod tests { (client, ids) } - fn readable(client: &mut StreamsState) -> Vec { - client - .events - .drain(..) - .filter_map(|event| match event { - StreamEvent::Readable { id } => Some(id), - _ => None, - }) - .collect() + /// A reader parked on each of a set of streams, to tell which of them a change woke + struct Readers(Vec<(StreamId, Counter, usize)>); + + impl Readers { + /// Park a reader on each of `ids` + fn park(client: &mut StreamsState, ids: &[StreamId]) -> Self { + let readers = Self(ids.iter().map(|&id| (id, Counter::new(), 0)).collect()); + readers.register(client); + readers + } + + fn register(&self, client: &mut StreamsState) { + for (id, waiter, _) in &self.0 { + client.waits.register(*id, Wait::Read, waiter); + } + } + + /// Drop the reader of `id`, as a reader that stops its stream goes away rather than wake to + /// its own stop + fn unpark(&mut self, id: StreamId) { + self.0.retain(|(parked, ..)| *parked != id); + } + + /// The streams whose reader woke since the last call, in the order they were parked, each + /// parked again + /// + /// Wakes count rather than order, so this cannot tell which of two woke first. + fn woken(&mut self, client: &mut StreamsState) -> Vec { + let woken = self + .0 + .iter_mut() + .filter_map(|(id, waiter, seen)| { + let wakes = waiter.wakes(); + (mem::replace(seen, wakes) != wakes).then_some(*id) + }) + .collect(); + self.register(client); + woken + } } /// Credit an eviction frees pays a shrunk window's debt first, so eviction goes on until the @@ -3751,18 +3774,19 @@ mod tests { VarInt::from_u64(3 * WINDOW).unwrap(), ); let ids = [0, 1, 2].map(|index| StreamId::new(Side::Server, Dir::Uni, index)); + let mut readers = Readers::park(&mut client, &ids); for id in ids { data(&mut client, id, 0, WINDOW as usize, false); assert_eq!(lease(&mut client, id), WINDOW as usize); } - assert_eq!(readable(&mut client), ids); + assert_eq!(readers.woken(&mut client), ids); let max_data = client.local_max_data; client.set_receive_window(VarInt::from_u64(3 * WINDOW / 2).unwrap()); let want = WINDOW as usize + 1; assert!(matches!(wait(&mut client, ids[0], want), Err(recv::Error::Blocked))); assert_eq!(client.credit_stats().evicted, 2); - assert_eq!(readable(&mut client), [ids[2], ids[1]], "newest first"); + assert_eq!(readers.woken(&mut client), [ids[1], ids[2]]); assert_eq!(client.local_max_data, max_data + WINDOW / 2); assert!(!client.leased(ids[0]).unwrap().evicted); } @@ -3873,6 +3897,7 @@ mod tests { VarInt::from_u64(3 * WINDOW).unwrap(), ); let ids = [0, 1, 2].map(|index| StreamId::new(Side::Server, Dir::Uni, index)); + let mut readers = Readers::park(&mut client, &ids); for id in ids { data(&mut client, id, 0, WINDOW as usize, false); } @@ -3889,13 +3914,13 @@ mod tests { )); } assert_eq!(client.credit_stats().evicted, 0); - readable(&mut client); + readers.woken(&mut client); // Its reader finishes a unit and parks; the credit only pays the debt. assert_eq!(lease(&mut client, ids[0]), WINDOW as usize); release(&mut client, ids[0], WINDOW as usize); assert_eq!(client.credit_stats().evicted, 1); - assert_eq!(readable(&mut client), [ids[2]]); + assert_eq!(readers.woken(&mut client), [ids[2]]); assert_eq!(client.local_max_data, max_data + WINDOW); } @@ -3944,6 +3969,7 @@ mod tests { VarInt::from_u64(3 * WINDOW).unwrap(), ); let ids = [0, 1, 2].map(|index| StreamId::new(Side::Server, Dir::Uni, index)); + let mut readers = Readers::park(&mut client, &ids); data(&mut client, ids[0], 0, WINDOW as usize, true); for id in [ids[1], ids[2]] { data(&mut client, id, 0, WINDOW as usize, false); @@ -3957,7 +3983,7 @@ mod tests { Err(recv::Error::Blocked) )); } - readable(&mut client); + readers.woken(&mut client); // The first reader leases the whole finished stream, then stops it; the credit only pays // the debt. @@ -3965,9 +3991,10 @@ mod tests { assert!(client.leased(ids[0]).unwrap().is_fin()); assert_eq!(client.credit_stats().evicted, 0); let mut pending = Retransmits::default(); + readers.unpark(ids[0]); assert!(recv(&mut client, &mut pending, ids[0]).stop(0u32.into()).is_ok()); assert_eq!(client.credit_stats().evicted, 1); - assert_eq!(readable(&mut client), [ids[2]]); + assert_eq!(readers.woken(&mut client), [ids[2]]); assert_eq!(client.local_max_data, max_data + WINDOW); } @@ -3985,6 +4012,7 @@ mod tests { VarInt::from_u64(3 * WINDOW).unwrap(), ); let ids = [0, 1, 2].map(|index| StreamId::new(Side::Server, Dir::Uni, index)); + let mut readers = Readers::park(&mut client, &ids); data(&mut client, ids[0], 0, 1, false); let _ = stop(&mut client, ids[0], 0); for id in [ids[1], ids[2]] { @@ -3996,12 +4024,12 @@ mod tests { )); } client.set_receive_window(VarInt::from_u64(2 * WINDOW).unwrap()); - readable(&mut client); + readers.woken(&mut client); // The peer spends the rest of its credit on the stopped stream; the debt absorbs it. data(&mut client, ids[0], 1, WINDOW as usize, false); assert_eq!(client.credit_stats().evicted, 1); - assert_eq!(readable(&mut client), [ids[2]], "newest first"); + assert_eq!(readers.woken(&mut client), [ids[2]], "newest first"); assert!(client.local_max_data > client.data_recvd, "the peer has credit"); } @@ -4080,19 +4108,20 @@ mod tests { fn evicted_taken_bytes_keep_credit_until_stop() { const WINDOW: u64 = 1000; let (mut client, ids) = stuck(WINDOW); + let mut readers = Readers::park(&mut client, &ids); let mut pending = Retransmits::default(); for id in ids { data(&mut client, id, 0, WINDOW as usize, false); assert_eq!(lease(&mut client, id), WINDOW as usize); let _ = recv(&mut client, &mut pending, id).take(WINDOW as usize); } - assert_eq!(readable(&mut client), ids); + assert_eq!(readers.woken(&mut client), ids); let max_data = client.local_max_data; // Debt as large as one stream's bytes, so the first stop only pays it. client.set_receive_window(VarInt::from_u64(WINDOW).unwrap()); assert!(matches!(wait(&mut client, ids[0], 1), Err(recv::Error::Blocked))); - assert_eq!(readable(&mut client), [ids[1]]); + assert_eq!(readers.woken(&mut client), [ids[1]]); assert_eq!(client.credit_stats().evicted, 1); assert_eq!(client.credit_stats().leased, 2 * WINDOW, "taken credit stays held"); assert!(matches!(wait(&mut client, ids[0], 1), Err(recv::Error::Blocked))); @@ -4105,6 +4134,7 @@ mod tests { assert_eq!(wait(&mut client, ids[1], 1), Err(recv::Error::Evicted)); // The stop pays the debt alone, so eviction resumes with the other stream. + readers.unpark(ids[1]); let _ = stop(&mut client, ids[1], 0); assert_eq!( recv(&mut client, &mut pending, ids[1]).received_reset(), @@ -4112,7 +4142,7 @@ mod tests { ); assert_eq!(client.local_max_data, max_data); assert_eq!(client.credit_stats().evicted, 2); - assert_eq!(readable(&mut client), [ids[0]]); + assert_eq!(readers.woken(&mut client), [ids[0]]); let _ = stop(&mut client, ids[0], 0); assert_eq!(client.credit_stats().leased, 0); assert_eq!(client.local_max_data, max_data + WINDOW); @@ -4125,12 +4155,13 @@ mod tests { fn stuck_window_evicts_the_lowest_priority() { const WINDOW: u64 = 1000; let (mut client, [low, high]) = stuck(WINDOW); + let mut readers = Readers::park(&mut client, &[low, high]); data(&mut client, high, 0, WINDOW as usize, false); data(&mut client, low, 0, WINDOW as usize - 1, false); let mut pending = Retransmits::default(); recv(&mut client, &mut pending, low).set_priority(-1); recv(&mut client, &mut pending, high).set_priority(1); - assert_eq!(readable(&mut client), [high, low]); + assert_eq!(readers.woken(&mut client), [low, high]); assert_eq!(lease(&mut client, high), WINDOW as usize); assert_eq!(lease(&mut client, low), WINDOW as usize - 1); assert!(matches!(wait(&mut client, high, 2000), Err(recv::Error::Blocked))); @@ -4143,11 +4174,11 @@ mod tests { data(&mut client, low, WINDOW - 1, 1, false); // The arrival woke the reader of `low`, which may yet consume. - assert_eq!(readable(&mut client), [low]); + assert_eq!(readers.woken(&mut client), [low]); assert!(!client.leased(low).unwrap().evicted); let blocked = wait_transmit(&mut client, low, 2000); assert_eq!(blocked, (Err(recv::Error::Blocked), ShouldTransmit(true))); - assert_eq!(readable(&mut client), [low], "the evicted reader wakes"); + assert_eq!(readers.woken(&mut client), [low], "the evicted reader wakes"); assert_eq!(wait(&mut client, low, 1), Err(recv::Error::Evicted)); assert_eq!(client.credit_stats().leased, WINDOW); assert_eq!(client.credit_stats().evicted, 1); @@ -4225,7 +4256,6 @@ mod tests { // Never leased again after this: its owner parked. assert_eq!(lease(&mut client, old), WINDOW as usize); data(&mut client, new, 0, WINDOW as usize, false); - let _ = readable(&mut client); assert!(matches!(wait(&mut client, new, 2000), Err(recv::Error::Blocked))); assert_eq!(wait(&mut client, new, 1), Err(recv::Error::Evicted)); assert_eq!(lease(&mut client, old), WINDOW as usize); @@ -4278,7 +4308,6 @@ mod tests { let mut pending = Retransmits::default(); let held = recv(&mut client, &mut pending, old).lease(0).0.unwrap(); data(&mut client, new, 0, WINDOW as usize, false); - let _ = readable(&mut client); assert!(matches!(wait(&mut client, new, 2000), Err(recv::Error::Blocked))); assert_eq!(client.credit_stats().evicted, 0); diff --git a/rs/moq-quic/src/connection/waits.rs b/rs/moq-quic/src/connection/waits.rs index 0ac10ccb045..3b27094e835 100644 --- a/rs/moq-quic/src/connection/waits.rs +++ b/rs/moq-quic/src/connection/waits.rs @@ -75,6 +75,9 @@ pub(crate) struct Waits { open: [WaiterList; 2], /// For a received datagram datagram: WaiterList, + /// For something to transmit: the driver, woken by whatever the application queues + transmit: WaiterList, + handshake_data: Latch, connected: Latch, confirmed: Latch, /// For the connection to be lost, which every other wait also hears @@ -141,6 +144,23 @@ impl Waits { self.datagram.wake(); } + /// Register `waiter` for something to transmit + pub(crate) fn register_transmit(&mut self, waiter: &Waiter) { + waiter.register(&mut self.transmit); + } + + /// The application queued something to transmit + /// + /// Cheap once woken: the list stays empty until the driver finds nothing to send again. + pub(crate) fn wake_transmit(&mut self) { + self.transmit.wake(); + } + + /// The handshake's negotiated parameters became available + pub(crate) fn handshake_data(&mut self) { + self.handshake_data.set(); + } + /// The handshake completed pub(crate) fn connected(&mut self) { self.connected.set(); @@ -151,6 +171,12 @@ impl Waits { self.confirmed.set(); } + /// Ready once the handshake's parameters are available, or with the error once the connection + /// was lost first + pub(crate) fn poll_handshake_data(&mut self, waiter: &Waiter) -> Poll> { + Self::poll_latch(&mut self.handshake_data, &self.lost, waiter) + } + /// Ready once the handshake completed, or with the error once the connection was lost first pub(crate) fn poll_connected(&mut self, waiter: &Waiter) -> Poll> { Self::poll_latch(&mut self.connected, &self.lost, waiter) @@ -205,6 +231,9 @@ impl Waits { list.wake(); } self.datagram.wake(); + // A close still owes the peer its CONNECTION_CLOSE. + self.transmit.wake(); + self.handshake_data.waiters.wake(); self.connected.waiters.wake(); self.confirmed.waiters.wake(); self.closed.wake(); diff --git a/rs/moq-quic/src/crypto.rs b/rs/moq-quic/src/crypto.rs index c110afe3e7e..8452aa20982 100644 --- a/rs/moq-quic/src/crypto.rs +++ b/rs/moq-quic/src/crypto.rs @@ -21,7 +21,7 @@ mod ring_like; #[cfg(any(feature = "aws-lc-rs", feature = "ring"))] pub(crate) mod rustls; -/// What the handshake negotiated, available once the connection emits `HandshakeDataReady`. +/// What the handshake negotiated, available once the connection's `poll_handshake_data` is ready. #[derive(Clone, Debug, PartialEq, Eq)] #[non_exhaustive] pub struct HandshakeData { @@ -45,7 +45,7 @@ pub(crate) trait Session: Send + Sync + 'static { /// Get data negotiated during the handshake, if available /// - /// Returns `None` until the connection emits `HandshakeDataReady`. + /// Returns `None` until the connection's `poll_handshake_data` is ready. fn handshake_data(&self) -> Option; /// The certificate chain the peer presented, end entity first, if any diff --git a/rs/moq-quic/src/fuzz/connection.rs b/rs/moq-quic/src/fuzz/connection.rs index 36312b4a87d..90ffe616bb3 100644 --- a/rs/moq-quic/src/fuzz/connection.rs +++ b/rs/moq-quic/src/fuzz/connection.rs @@ -10,6 +10,7 @@ use std::{ collections::{HashMap, HashSet, VecDeque}, net::{Ipv6Addr, SocketAddr}, num::NonZeroU64, + task::Poll, }; use arbitrary::{Arbitrary, Unstructured}; @@ -17,7 +18,7 @@ use bytes::{Buf, Bytes, BytesMut}; use super::plaintext::{Clear, Plaintext, TAG_LEN}; use crate::{ - Dir, Duration, Event, Instant, Limits, Side, StreamId, + Dir, Duration, Instant, Limits, Side, StreamId, connection::Connection, endpoint::{self, ConnectionHandle, DatagramEvent, Dial, Endpoint, Received}, frame::{self, Frame}, @@ -483,7 +484,7 @@ impl Pair { conn.handle_event(event); } } - while let Some(transmit) = conn.poll_transmit(now, MAX_DATAGRAMS, &mut buf) { + while let Some(transmit) = conn.transmit(now, MAX_DATAGRAMS, &mut buf) { progress = true; send( &mut other.inbound, @@ -498,20 +499,15 @@ impl Pair { } } - while let Some(event) = conn.poll() { - match event { - Event::Connected => peer.connected = true, - Event::DatagramReceived => { - while let Some(datagram) = conn.datagrams().recv() { - if peer.forged_datagrams.contains(&datagram) { - continue; - } - let seq = check_datagram(&datagram, &other.datagrams_sent); - assert!(peer.datagrams_received.insert(seq), "datagram {seq} delivered twice"); - } - } - _ => {} + // Nothing waits between polls, so each wait is only asked for its answer as it stands. + let waiter = kio::Waiter::noop(); + peer.connected |= conn.poll_connected(&waiter) == Poll::Ready(Ok(())); + while let Poll::Ready(Ok(datagram)) = conn.datagrams().poll_recv(&waiter) { + if peer.forged_datagrams.contains(&datagram) { + continue; } + let seq = check_datagram(&datagram, &other.datagrams_sent); + assert!(peer.datagrams_received.insert(seq), "datagram {seq} delivered twice"); } } diff --git a/rs/moq-quic/src/lib.rs b/rs/moq-quic/src/lib.rs index a84e1c42d06..da03c076484 100644 --- a/rs/moq-quic/src/lib.rs +++ b/rs/moq-quic/src/lib.rs @@ -53,7 +53,7 @@ use bloom_token_log::BloomTokenLog; pub mod connection; pub use crate::connection::{ - CreditStats, Datagrams, Event, FrameStats, PathStats, ShouldTransmit, StreamEvent, Streams, UdpStats, recv, send, + CreditStats, Datagrams, Event, FrameStats, PathStats, ShouldTransmit, Streams, UdpStats, recv, send, }; pub mod stream; diff --git a/rs/moq-quic/src/tests/acked.rs b/rs/moq-quic/src/tests/acked.rs index 7be6b220f92..75fd6fcf850 100644 --- a/rs/moq-quic/src/tests/acked.rs +++ b/rs/moq-quic/src/tests/acked.rs @@ -1,6 +1,10 @@ -//! Tests for [`send::Stream::acked`], which reports whether the peer acknowledged a send stream's prefix +//! Tests for [`send::Stream::acked`] and [`send::Stream::poll_acked`], which report whether the +//! peer acknowledged a send stream's prefix + +use std::task::Poll; use super::*; +use crate::connection::waits::Counter; const LATENCY: Duration = Duration::from_millis(10); @@ -31,17 +35,14 @@ fn acked(pair: &mut Pair, ch: ConnectionHandle, s: StreamId, offset: send::Offse pair.client_send(ch, s).acked(offset) } -/// The prefix the client last reported acknowledged on `s`, draining its events -fn reported(pair: &mut Pair, ch: ConnectionHandle, s: StreamId) -> Option { - let mut reported = None; - while let Some(event) = pair.client_conn_mut(ch).poll() { - if let Event::Stream(StreamEvent::Acked { id, offset }) = event - && id == s - { - reported = Some(offset); - } - } - reported +fn poll_acked( + pair: &mut Pair, + ch: ConnectionHandle, + s: StreamId, + waiter: &kio::Waiter, + offset: send::Offset, +) -> Poll> { + pair.client_send(ch, s).poll_acked(waiter, offset) } /// Transmit the client's pending packets and drop them @@ -51,8 +52,7 @@ fn lose_client_packets(pair: &mut Pair) { pair.client.outbound.clear(); } -/// An offset is acknowledged only once every byte before it is, and the event that reports it -/// fires only for an offset someone polled. +/// An offset is acknowledged only once every byte before it is, and only that wakes its waiter. #[test] fn waits_for_contiguous_prefix() { let _guard = subscribe(); @@ -60,7 +60,6 @@ fn waits_for_contiguous_prefix() { let (client_ch, server_ch) = pair.connect(); let s = open(&mut pair, client_ch, server_ch); pair.drive(); - reported(&mut pair, client_ch, s); pair.client_send(client_ch, s).write(&mut &b"lost"[..]).unwrap(); let lost = pair.client_send(client_ch, s).offset(); @@ -69,22 +68,26 @@ fn waits_for_contiguous_prefix() { // Acknowledged, but behind the lost packet: the middle of the stream, not its prefix pair.client_send(client_ch, s).write(&mut &b"after"[..]).unwrap(); let after = pair.client_send(client_ch, s).offset(); + let waiter = Counter::new(); + assert!(poll_acked(&mut pair, client_ch, s, &waiter, after).is_pending()); let acks = pair.client_conn_mut(client_ch).stats().frame_rx.acks; while pair.client_conn_mut(client_ch).stats().frame_rx.acks == acks { assert!(pair.step(), "never acknowledged"); } + assert_eq!(waiter.wakes(), 0); assert_eq!(acked(&mut pair, client_ch, s, after), Ok(false)); // The retransmission fills the gap, which acknowledges both at once pair.drive(); - assert_eq!(reported(&mut pair, client_ch, s), Some(b"headerlostafter".len() as u64)); + assert_eq!(waiter.wakes(), 1); + assert_eq!(poll_acked(&mut pair, client_ch, s, &waiter, after), Poll::Ready(Ok(()))); assert_eq!(acked(&mut pair, client_ch, s, lost), Ok(true)); assert_eq!(acked(&mut pair, client_ch, s, after), Ok(true)); assert!(pair.client_conn_mut(client_ch).stats().path.lost_packets > 0); } -/// A waiter past the offset an event reports polls again and registers anew, so it learns of -/// a later ACK too. +/// A waiter past the offset the acknowledged prefix reached wakes with the rest, polls again and +/// registers anew, so it learns of a later ACK too. #[test] fn later_waiter_registers_again() { let _guard = subscribe(); @@ -92,7 +95,6 @@ fn later_waiter_registers_again() { let (client_ch, server_ch) = pair.connect(); let s = open(&mut pair, client_ch, server_ch); pair.drive(); - reported(&mut pair, client_ch, s); // The first packet is held, and the second lost pair.client_send(client_ch, s).write(&mut &b"first"[..]).unwrap(); @@ -102,23 +104,33 @@ fn later_waiter_registers_again() { pair.client_send(client_ch, s).write(&mut &b"second"[..]).unwrap(); let second = pair.client_send(client_ch, s).offset(); lose_client_packets(&mut pair); - assert_eq!(acked(&mut pair, client_ch, s, second), Ok(false)); - assert_eq!(acked(&mut pair, client_ch, s, first), Ok(false)); + let (first_waiter, second_waiter) = (Counter::new(), Counter::new()); + assert!(poll_acked(&mut pair, client_ch, s, &second_waiter, second).is_pending()); + assert!(poll_acked(&mut pair, client_ch, s, &first_waiter, first).is_pending()); // Only the first arrives at first pair.client.finish_delay(); - while acked(&mut pair, client_ch, s, first) != Ok(true) { + while first_waiter.wakes() == 0 { assert!(pair.step(), "never acknowledged"); } - assert_eq!(reported(&mut pair, client_ch, s), Some(u64::from(first))); - assert_eq!(acked(&mut pair, client_ch, s, second), Ok(false)); + assert_eq!( + poll_acked(&mut pair, client_ch, s, &first_waiter, first), + Poll::Ready(Ok(())) + ); + assert_eq!(second_waiter.wakes(), 1); + assert!(poll_acked(&mut pair, client_ch, s, &second_waiter, second).is_pending()); - // The retransmission reports the second + // The retransmission wakes the second again pair.drive(); - assert_eq!(reported(&mut pair, client_ch, s), Some(u64::from(second))); + assert_eq!(second_waiter.wakes(), 2); + assert_eq!( + poll_acked(&mut pair, client_ch, s, &second_waiter, second), + Poll::Ready(Ok(())) + ); } -/// Without a poll waiting on it, an acknowledgment is not reported. +/// Without a poll waiting on it, an acknowledgment leaves no waiter behind, and a later poll +/// still finds it. #[test] fn unpolled_is_not_reported() { let _guard = subscribe(); @@ -129,7 +141,7 @@ fn unpolled_is_not_reported() { pair.client_send(client_ch, s).write(&mut &b"payload"[..]).unwrap(); let offset = pair.client_send(client_ch, s).offset(); pair.drive(); - assert_eq!(reported(&mut pair, client_ch, s), None); + assert_eq!(pair.client_conn_mut(client_ch).waiting_streams(), 0); assert_eq!(acked(&mut pair, client_ch, s, offset), Ok(true)); } @@ -153,8 +165,8 @@ fn already_acked() { assert_eq!(acked(&mut pair, client_ch, s, all), Ok(true)); } -/// A finished stream is freed by the ACK that completes it, which reports a waiting offset -/// before the stream finishes. +/// A finished stream is freed by the ACK that completes it, which wakes a waiting offset as well +/// as the end. #[test] fn finished_stream_reports_before_freed() { let _guard = subscribe(); @@ -162,26 +174,24 @@ fn finished_stream_reports_before_freed() { let (client_ch, server_ch) = pair.connect(); let s = open(&mut pair, client_ch, server_ch); pair.drive(); - reported(&mut pair, client_ch, s); pair.client_send(client_ch, s).write(&mut &b"payload"[..]).unwrap(); let offset = pair.client_send(client_ch, s).offset(); - assert_eq!(acked(&mut pair, client_ch, s, offset), Ok(false)); + let (acked_waiter, end) = (Counter::new(), Counter::new()); + assert!(poll_acked(&mut pair, client_ch, s, &acked_waiter, offset).is_pending()); pair.client_send(client_ch, s).finish().unwrap(); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); pair.drive(); assert_eq!(pair.client_streams(client_ch).send_streams(), 0); - let mut events = Vec::new(); - while let Some(Event::Stream(event)) = pair.client_conn_mut(client_ch).poll() { - events.push(event); - } - let len = b"headerpayload".len() as u64; + assert_eq!((acked_waiter.wakes(), end.wakes()), (1, 1)); assert_eq!( - events, - [ - StreamEvent::Acked { id: s, offset: len }, - StreamEvent::Finished { id: s } - ] + poll_acked(&mut pair, client_ch, s, &acked_waiter, offset), + Poll::Ready(Ok(())) + ); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); // The freed stream's end is kept until released, so the offset is still answered. assert_eq!(acked(&mut pair, client_ch, s, offset), Ok(true)); @@ -191,36 +201,62 @@ fn finished_stream_reports_before_freed() { assert_eq!(pair.client_send(client_ch, s).end(), Err(send::Error::Closed)); } -/// A reset fails the offsets past its Reliable Size at once, while those before it are -/// still acknowledged once the prefix arrives. -#[test] -fn reset_fails_past_reliable_size() { - let _guard = subscribe(); - let mut pair = pair(true); - let (client_ch, server_ch) = pair.connect(); - let s = open(&mut pair, client_ch, server_ch); - +/// Opens a stream whose written prefix and the rest are both lost, resets it after the prefix, +/// and parks a waiter on the prefix, returning the stream, the prefix's end, and the waiter +fn reset_after_prefix( + pair: &mut Pair, + client_ch: ConnectionHandle, + server_ch: ConnectionHandle, +) -> (StreamId, send::Offset, Counter) { + let s = open(pair, client_ch, server_ch); pair.client_send(client_ch, s).write(&mut &b"kept"[..]).unwrap(); let kept = pair.client_send(client_ch, s).offset(); let reset = pair.client_send(client_ch, s).offset(); pair.client_send(client_ch, s).write(&mut &b"dropped"[..]).unwrap(); let dropped = pair.client_send(client_ch, s).offset(); - lose_client_packets(&mut pair); - reported(&mut pair, client_ch, s); + lose_client_packets(pair); pair.client_send(client_ch, s).reset_at(reset, VarInt(42)).unwrap(); - assert_eq!(acked(&mut pair, client_ch, s, dropped), Err(send::Error::Closed)); - assert_eq!(acked(&mut pair, client_ch, s, kept), Ok(false)); - assert_eq!(reported(&mut pair, client_ch, s), None); + let waiter = Counter::new(); + assert_eq!( + poll_acked(pair, client_ch, s, &waiter, dropped), + Poll::Ready(Err(send::Error::Closed)) + ); + assert!(poll_acked(pair, client_ch, s, &waiter, kept).is_pending()); + (s, kept, waiter) +} + +/// A reset fails the offsets past its Reliable Size at once, while a waiter before it still +/// wakes once the prefix arrives. +#[test] +fn reset_fails_past_reliable_size() { + let _guard = subscribe(); + let mut pair = pair(true); + let (client_ch, server_ch) = pair.connect(); + let (_, _, waiter) = reset_after_prefix(&mut pair, client_ch, server_ch); - // The retransmitted prefix is acknowledged before the reset frees the stream + // The retransmitted prefix is acknowledged by the ACK that frees the stream pair.drive(); assert_eq!(pair.client_streams(client_ch).send_streams(), 0); - assert_eq!(reported(&mut pair, client_ch, s), Some(b"headerkept".len() as u64)); + assert_eq!(waiter.wakes(), 1); } -/// A waiter whose offset a later reset drops learns so from the event, even beside a lower -/// waiter the reset keeps. +/// The waiter a reset's acknowledged prefix wakes learns it was acknowledged, even though the +/// same ACK freed the stream. +#[test] +fn reset_keeps_its_acked_prefix() { + let _guard = subscribe(); + let mut pair = pair(true); + let (client_ch, server_ch) = pair.connect(); + let (s, kept, waiter) = reset_after_prefix(&mut pair, client_ch, server_ch); + + pair.drive(); + assert_eq!(waiter.wakes(), 1); + assert_eq!(poll_acked(&mut pair, client_ch, s, &waiter, kept), Poll::Ready(Ok(()))); +} + +/// A waiter whose offset a later reset drops wakes to learn so, even beside a lower waiter the +/// reset keeps. #[test] fn reset_reports_a_dropped_waiter() { let _guard = subscribe(); @@ -234,18 +270,21 @@ fn reset_reports_a_dropped_waiter() { pair.client_send(client_ch, s).write(&mut &b"dropped"[..]).unwrap(); let dropped = pair.client_send(client_ch, s).offset(); lose_client_packets(&mut pair); - reported(&mut pair, client_ch, s); - assert_eq!(acked(&mut pair, client_ch, s, kept), Ok(false)); - assert_eq!(acked(&mut pair, client_ch, s, dropped), Ok(false)); + let (kept_waiter, dropped_waiter) = (Counter::new(), Counter::new()); + assert!(poll_acked(&mut pair, client_ch, s, &kept_waiter, kept).is_pending()); + assert!(poll_acked(&mut pair, client_ch, s, &dropped_waiter, dropped).is_pending()); pair.client_send(client_ch, s).reset_at(kept, VarInt(42)).unwrap(); assert_eq!(pair.client_send(client_ch, s).offset(), kept); - assert!(reported(&mut pair, client_ch, s).is_some()); - assert_eq!(acked(&mut pair, client_ch, s, dropped), Err(send::Error::Closed)); + assert_eq!(dropped_waiter.wakes(), 1); + assert_eq!( + poll_acked(&mut pair, client_ch, s, &dropped_waiter, dropped), + Poll::Ready(Err(send::Error::Closed)) + ); } /// A peer that stopped reading will not read what it acknowledges, before or after the -/// offset was taken, and a waiter learns so from the event. +/// offset was taken, and a waiter wakes to learn so. #[test] fn stop_sending_fails() { let _guard = subscribe(); @@ -256,7 +295,8 @@ fn stop_sending_fails() { pair.client_send(client_ch, s).write(&mut &b"unread"[..]).unwrap(); let before = pair.client_send(client_ch, s).offset(); lose_client_packets(&mut pair); - assert_eq!(acked(&mut pair, client_ch, s, before), Ok(false)); + let waiter = Counter::new(); + assert!(poll_acked(&mut pair, client_ch, s, &waiter, before).is_pending()); pair.server_recv(server_ch, s).stop(VarInt(7)).unwrap(); pair.drive(); @@ -265,7 +305,11 @@ fn stop_sending_fails() { Ok(Some(send::End::Stopped(VarInt(7)))) ); let after = pair.client_send(client_ch, s).offset(); - assert_eq!(reported(&mut pair, client_ch, s), Some(b"header".len() as u64)); + assert_eq!(waiter.wakes(), 1); + assert_eq!( + poll_acked(&mut pair, client_ch, s, &waiter, before), + Poll::Ready(Err(send::Error::Closed)) + ); for offset in [before, after] { assert_eq!(acked(&mut pair, client_ch, s, offset), Err(send::Error::Closed)); } diff --git a/rs/moq-quic/src/tests/characterize/frames.rs b/rs/moq-quic/src/tests/characterize/frames.rs index 008383462a7..ceb9f02c7f9 100644 --- a/rs/moq-quic/src/tests/characterize/frames.rs +++ b/rs/moq-quic/src/tests/characterize/frames.rs @@ -299,14 +299,17 @@ fn retire_connection_id_in_use() { } /// A connection closed by its own transport error sends CONNECTION_CLOSE to a CID the server -/// retired, and the server answers with a stateless reset, which must neither replace the unread -/// close reason with `Reset` nor, once the reason was read, report a second `ConnectionLost`. +/// retired, and the server answers with a stateless reset, which must neither replace the close +/// reason with `Reset`, whether or not it was read first, nor wake the closed wait again. fn stateless_reset_after_local_close(poll_early: bool) { let _guard = subscribe(); let (mut pair, client_ch, server_ch) = connected(); retire_in_use(&mut pair, client_ch, server_ch); + let closed = Counter::new(); + assert!(pair.client_conn_mut(client_ch).poll_closed(&closed).is_pending()); pair.drive_client(); if poll_early { + assert_eq!(closed.wakes(), 1); assert_eq!( lost_code(pair.client_conn_mut(client_ch)), TransportErrorCode::CONNECTION_ID_LIMIT_ERROR @@ -314,18 +317,14 @@ fn stateless_reset_after_local_close(poll_early: bool) { } pair.drive(); - let lost = lost(pair.client_conn_mut(client_ch)); - if poll_early { - assert_eq!(lost, None); - } else { - assert_matches!( - lost, - Some(ConnectionError::TransportError(TransportError { - code: TransportErrorCode::CONNECTION_ID_LIMIT_ERROR, - .. - })) - ); - } + assert_eq!(closed.wakes(), 1); + assert_matches!( + lost(pair.client_conn_mut(client_ch)), + Some(ConnectionError::TransportError(TransportError { + code: TransportErrorCode::CONNECTION_ID_LIMIT_ERROR, + .. + })) + ); } #[test] diff --git a/rs/moq-quic/src/tests/characterize/mod.rs b/rs/moq-quic/src/tests/characterize/mod.rs index 504380f6b0d..b18d178d50b 100644 --- a/rs/moq-quic/src/tests/characterize/mod.rs +++ b/rs/moq-quic/src/tests/characterize/mod.rs @@ -107,29 +107,17 @@ fn server_sends(pair: &mut Pair, ch: ConnectionHandle, space: SpaceId, frames: F .push_back((pair.time, None, packet.as_slice().into())); } -/// Drains `conn`'s events, returning the error it was lost with, if any +/// The error `conn` was lost with, if any fn lost(conn: &mut Connection) -> Option { - let mut lost = None; - while let Some(event) = conn.poll() { - if let Event::ConnectionLost { reason } = event { - assert!(lost.is_none(), "lost twice: {lost:?}, then {reason:?}"); - lost = Some(reason); - } - } - lost + conn.error().cloned() } -/// Drains `conn`'s events, panicking unless it connected and was never lost +/// Panics unless `conn` connected and was never lost fn assert_connected(conn: &mut Connection) { - let mut connected = false; - while let Some(event) = conn.poll() { - match event { - Event::Connected => connected = true, - Event::ConnectionLost { reason } => panic!("lost: {reason:?}"), - _ => {} - } + if let Some(reason) = conn.error() { + panic!("lost: {reason:?}"); } - assert!(connected, "never connected"); + assert!(is_connected(conn), "never connected"); } /// The transport error code `conn` was lost with, panicking if it is still up or lost otherwise diff --git a/rs/moq-quic/src/tests/mod.rs b/rs/moq-quic/src/tests/mod.rs index 5de98e4bc1f..ad88076dee1 100644 --- a/rs/moq-quic/src/tests/mod.rs +++ b/rs/moq-quic/src/tests/mod.rs @@ -9,6 +9,7 @@ use std::{ Arc, Mutex, atomic::{AtomicU64, Ordering}, }, + task::Poll, }; use assert_matches::assert_matches; @@ -31,7 +32,7 @@ use crate::{ Duration, Instant, cid_generator::{ConnectionIdGenerator, RandomConnectionIdGenerator}, congestion::{self, Controller, Factory, Metrics}, - connection::{Connection, Error as ConnectionError}, + connection::{Connection, Error as ConnectionError, waits::Counter}, endpoint::{Accepting, ConnectionHandle, DatagramEvent, Dial, Endpoint, Incoming, Received}, frame::FrameStruct, packet::{Header, InitialHeader, PacketNumber}, @@ -94,8 +95,7 @@ fn pending_incoming_survives_server_config_change() { let server_ch = pair.server.try_accept(incoming, pair.time).unwrap(); pair.drive(); for (endpoint, ch) in [(&mut pair.client, client_ch), (&mut pair.server, server_ch)] { - let conn = endpoint.connections.get_mut(&ch).unwrap(); - assert!(iter::from_fn(|| conn.poll()).any(|event| matches!(event, Event::Connected))); + assert!(is_connected(endpoint.connections.get_mut(&ch).unwrap())); } } } @@ -120,7 +120,7 @@ fn pending_incoming_can_retry_after_disabling_server() { }); pair.drive(); pair.server.assert_accept(); - assert!(iter::from_fn(|| pair.client_conn_mut(client_ch).poll()).any(|event| matches!(event, Event::Connected))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } #[test] @@ -228,12 +228,7 @@ fn version_negotiate_client() { if let Some(DatagramEvent::ConnectionEvent(_, event)) = opt_event { client_ch.handle_event(event); } - assert_matches!( - client_ch.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::VersionMismatch, - }) - ); + assert_matches!(client_ch.error(), Some(ConnectionError::VersionMismatch)); } #[test] @@ -241,11 +236,14 @@ fn lifecycle() { let _guard = subscribe(); let mut pair = Pair::default(); let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + assert_matches!(pair.client_conn_mut(client_ch).error(), None); assert!(pair.client_conn_mut(client_ch).using_ecn()); assert!(pair.server_conn_mut(server_ch).using_ecn()); const REASON: &str = "whee"; + let (server_closed, client_closed) = (Counter::new(), Counter::new()); + assert!(pair.server_conn_mut(server_ch).poll_closed(&server_closed).is_pending()); + assert!(pair.client_conn_mut(client_ch).poll_closed(&client_closed).is_pending()); info!("closing"); pair.client .connections @@ -253,15 +251,15 @@ fn lifecycle() { .unwrap() .close(pair.time, VarInt(42), REASON.into()); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ApplicationClosed( + assert_eq!((server_closed.wakes(), client_closed.wakes()), (1, 1)); + assert_matches!(pair.server_conn_mut(server_ch).error(), + Some(ConnectionError::ApplicationClosed( ApplicationClose { error_code: VarInt(42), reason } - )}) if reason == REASON); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::LocallyClosed { + )) if reason == REASON); + assert_matches!(pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::LocallyClosed { code: VarInt(42), reason } - }) if reason == REASON); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + ) if reason == REASON); assert_eq!(pair.client.known_connections(), 0); assert_eq!(pair.client.known_cids(), 0); assert_eq!(pair.server.known_connections(), 0); @@ -321,8 +319,8 @@ fn connect_before_the_finished_leaves(pair: &mut Pair, config: client::Config) - } let client = pair.client_conn_mut(client_ch); - assert_matches!(client.poll(), Some(Event::HandshakeDataReady)); - assert_matches!(client.poll(), Some(Event::Connected)); + assert!(has_handshake_data(client)); + assert!(is_connected(client)); client_ch } @@ -380,7 +378,7 @@ fn close_before_a_client_certificate_leaves_ends() { let mut transmits = Vec::new(); let mut buf = Vec::new(); while transmits.len() <= 2 - && let Some(transmit) = client.poll_transmit(now, 1, &mut buf) + && let Some(transmit) = client.transmit(now, 1, &mut buf) { let datagram = Bytes::copy_from_slice(&buf[..transmit.size]); transmits.push((transmit, datagram)); @@ -392,21 +390,15 @@ fn close_before_a_client_certificate_leaves_ends() { let server_ch = pair.server.assert_accept(); let server = pair.server_conn_mut(server_ch); - let mut connected = false; - while let Some(event) = server.poll() { - match event { - Event::Connected => connected = true, - Event::ConnectionLost { reason } if connected => { - assert_matches!( - reason, - ConnectionError::ApplicationClosed(ApplicationClose { - error_code: VarInt(42), - .. - }) - ); - } - _ => {} - } + let connected = is_connected(server); + if connected && let Some(reason) = server.error() { + assert_matches!( + reason, + ConnectionError::ApplicationClosed(ApplicationClose { + error_code: VarInt(42), + .. + }) + ); } if connected { carried += 1; @@ -441,13 +433,13 @@ fn close_before_the_server_has_the_finished(lost: LostFlight) { let server_ch = pair.server.assert_accept(); let server = pair.server_conn_mut(server_ch); - assert_matches!(server.poll(), Some(Event::HandshakeDataReady)); - assert_matches!(server.poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(server.poll(), Some(Event::Connected)); - assert_matches!(server.poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ApplicationClosed( + assert!(has_handshake_data(server)); + assert!(is_confirmed(server)); + assert!(is_connected(server)); + assert_matches!(server.error(), + Some(ConnectionError::ApplicationClosed( ApplicationClose { error_code: VarInt(42), reason } - )}) if reason == REASON); + )) if reason == REASON); } #[test] @@ -548,12 +540,7 @@ fn server_stateless_reset() { assert!(!pair.client.endpoint.is_stateless_reset(pair.server.addr, &[0x40; 64])); assert!(!pair.client.endpoint.is_stateless_reset(pair.client.addr, reset)); pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::Reset - }) - ); + assert_matches!(pair.client_conn_mut(client_ch).error(), Some(ConnectionError::Reset)); } /// A reset answering a locally closed connection drains it without reporting a loss beyond the @@ -593,12 +580,9 @@ fn client_stateless_reset() { let conn = pair.server_conn_mut(server_ch); assert!(conn.is_drained()); assert_matches!( - conn.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::LocallyClosed { code: VarInt(42), .. } - }) + conn.error(), + Some(ConnectionError::LocallyClosed { code: VarInt(42), .. }) ); - assert_matches!(conn.poll(), None); } /// Restart the server from a fresh config built from `restart`, and report whether the client's @@ -623,12 +607,7 @@ fn restarted_server_resets(restart: &[u8; 32]) -> bool { .unwrap(); pair.client.connections.get_mut(&client_ch).unwrap().ping(); pair.drive(); - matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::Reset - }) - ) + matches!(pair.client_conn_mut(client_ch).error(), Some(ConnectionError::Reset)) } #[test] @@ -723,11 +702,10 @@ fn stateless_reset_while_draining_keeps_close_reason() { assert!(pair.client_conn_mut(client_ch).is_drained()); assert_eq!(pair.client_conn_mut(client_ch).poll_timeout(), None); assert_eq!(pair.client.known_connections(), 0); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ApplicationClosed( + assert_matches!(pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ApplicationClosed( ApplicationClose { error_code: VarInt(42), reason } - )}) if reason == REASON); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + )) if reason == REASON); } /// The application consumes the peer's close reason before the reset is processed. @@ -740,11 +718,14 @@ fn stateless_reset_while_draining_is_not_a_second_loss() { const REASON: &str = "not accepted"; server_closes_and_forgets(&mut pair, server_ch, REASON); + let closed = Counter::new(); + assert!(pair.client_conn_mut(client_ch).poll_closed(&closed).is_pending()); pair.drive_client(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ApplicationClosed( + assert_eq!(closed.wakes(), 1); + assert_matches!(pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ApplicationClosed( ApplicationClose { error_code: VarInt(42), reason } - )}) if reason == REASON); + )) if reason == REASON); pair.drive_server(); assert!( @@ -756,7 +737,15 @@ fn stateless_reset_while_draining_is_not_a_second_loss() { assert!(pair.client_conn_mut(client_ch).is_drained()); assert_eq!(pair.client_conn_mut(client_ch).poll_timeout(), None); assert_eq!(pair.client.known_connections(), 0); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + // The reset neither wakes the closed wait again nor replaces the reason. + assert_eq!(closed.wakes(), 1); + assert_matches!( + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ApplicationClosed(ApplicationClose { + error_code: VarInt(42), + .. + })) + ); } /// Process CONNECTION_CLOSE followed by two resets in one batch. The first reset ends draining; @@ -790,11 +779,10 @@ fn stateless_reset_after_draining_keeps_close_reason() { assert!(pair.client_conn_mut(client_ch).is_drained()); assert_eq!(pair.client.known_connections(), 0); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ApplicationClosed( + assert_matches!(pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ApplicationClosed( ApplicationClose { error_code: VarInt(42), reason } - )}) if reason == REASON); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + )) if reason == REASON); } /// Leave the server's CONNECTION_CLOSE unread until the server discards its connection state. @@ -931,27 +919,26 @@ fn finish_stream_simple() { pair.client_send(client_ch, s).write(&mut &MSG[..]).unwrap(); assert_eq!(pair.client_streams(client_ch).send_streams(), 1); pair.client_send(client_ch, s).finish().unwrap(); + let (end, accept) = (Counter::new(), Counter::new()); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Uni) + .is_pending() + ); pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Finished { id })) if id == s + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); assert_eq!(pair.client_streams(client_ch).send_streams(), 0); assert_eq!(pair.server_conn_mut(client_ch).streams().send_streams(), 0); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Readable { id })) if id == s - ); - // Receive-only streams do not get `StreamFinished` events + assert_eq!(accept.wakes(), 1); + // Receive-only streams hold no send state assert_eq!(pair.server_conn_mut(client_ch).streams().send_streams(), 0); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); let mut recv = pair.server_recv(server_ch, s); let mut chunks = recv.read().unwrap(); @@ -983,16 +970,16 @@ fn reset_stream() { pair.client_send(client_ch, s).reset(ERROR).unwrap(); pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); let mut recv = pair.server_recv(server_ch, s); let mut chunks = recv.read().unwrap(); assert_matches!(chunks.next(usize::MAX), Err(recv::Error::Reset(ERROR))); let _ = chunks.finalize(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + // The reset stream ends reset, never finished. + assert_eq!( + peek(|waiter| pair.client_send(client_ch, s).poll_end(waiter)), + Poll::Ready(Ok(send::End::Reset(ERROR))) + ); } #[test] @@ -1014,10 +1001,6 @@ fn stop_stream() { pair.server_recv(server_ch, s).stop(ERROR).unwrap(); pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); assert_matches!( @@ -1048,8 +1031,8 @@ fn reject_self_signed_server_cert() { pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::TransportError(error)}) + assert_matches!(pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::TransportError(error)) if error.code == TransportErrorCode::crypto(AlertDescription::UnknownCA.into())); } @@ -1084,17 +1067,20 @@ fn reject_missing_client_cert() { pair.drive(); // The client completes the connection, but finds it immediately closed - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ConnectionClosed(close)}) + let client = pair.client_conn_mut(client_ch); + assert!(has_handshake_data(client)); + assert!(is_connected(client)); + assert_matches!(client.error(), + Some(ConnectionError::ConnectionClosed(close)) if close.error_code == TransportErrorCode::crypto(AlertDescription::CertificateRequired.into())); // The server never completes the connection let server_ch = pair.server.assert_accept(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::TransportError(error)}) + let server = pair.server_conn_mut(server_ch); + assert!(has_handshake_data(server)); + assert!(!is_connected(server)); + assert_matches!(server.error(), + Some(ConnectionError::TransportError(error)) if error.code == TransportErrorCode::crypto(AlertDescription::CertificateRequired.into())); } @@ -1357,9 +1343,7 @@ fn alpn_success() { let client_ch = pair.begin_connect(client_config); pair.drive(); let server_ch = pair.server.assert_accept(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::Connected)); + assert_established(pair.server_conn_mut(server_ch)); let client = pair.client_conn_mut(client_ch).handshake_data().unwrap(); assert_eq!(client.alpn.as_deref(), Some(&b"bar"[..])); @@ -1378,8 +1362,8 @@ fn server_alpn_unset() { let client_ch = pair.begin_connect(client_config); pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ConnectionClosed(err) }) if err.error_code == TransportErrorCode::crypto(0x78) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ConnectionClosed(err)) if err.error_code == TransportErrorCode::crypto(0x78) ); } @@ -1396,8 +1380,8 @@ fn client_alpn_unset() { let client_ch = pair.begin_connect(client_config()); pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ConnectionClosed(err) }) if err.error_code == TransportErrorCode::crypto(0x78) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ConnectionClosed(err)) if err.error_code == TransportErrorCode::crypto(0x78) ); } @@ -1418,8 +1402,8 @@ fn alpn_mismatch() { pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ConnectionClosed(err) }) if err.error_code == TransportErrorCode::crypto(0x78) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ConnectionClosed(err)) if err.error_code == TransportErrorCode::crypto(0x78) ); } @@ -1438,13 +1422,14 @@ fn late_peer_params_make_streams_available() { pair.server.drive_incoming(pair.time, pair.client.addr); let incoming = pair.server.pop_waiting_incoming(); let server_ch = pair.server.try_accept(incoming, pair.time).unwrap(); - assert_eq!(pair.server_streams(server_ch).open(Dir::Uni, Default::default()), None); + let open = Counter::new(); + let poll = pair + .server_streams(server_ch) + .poll_open(&open, Dir::Uni, Default::default()); + assert!(poll.is_pending()); pair.drive(); - assert!( - iter::from_fn(|| pair.server_conn_mut(server_ch).poll()) - .any(|event| matches!(event, Event::Stream(StreamEvent::Available { dir: Dir::Uni }))) - ); + assert_eq!(open.wakes(), 1); assert!( pair.server_streams(server_ch) .open(Dir::Uni, Default::default()) @@ -1485,24 +1470,25 @@ fn stream_id_limit() { const MSG: &[u8] = b"hello"; pair.client_send(client_ch, s).write(&mut &MSG[..]).unwrap(); pair.client_send(client_ch, s).finish().unwrap(); + let (end, accept) = (Counter::new(), Counter::new()); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Uni) + .is_pending() + ); pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Finished { id })) if id == s + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); assert_eq!( pair.client_streams(client_ch).open(Dir::Uni, Default::default()), None, "server does not immediately grant additional credit" ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Readable { id })) if id == s - ); + assert_eq!(accept.wakes(), 1); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); let mut recv = pair.server_recv(server_ch, s); @@ -1520,19 +1506,15 @@ fn stream_id_limit() { pair.to_client.loss = 1.0; drive_until(&mut pair, |pair| grants(pair) > before); let lost = grants(&mut pair); - assert_eq!( - pair.client_streams(client_ch).open(Dir::Uni, Default::default()), - None, - "the dropped grant did not reach the sender" - ); + let open = Counter::new(); + let poll = pair + .client_streams(client_ch) + .poll_open(&open, Dir::Uni, Default::default()); + assert!(poll.is_pending(), "the dropped grant did not reach the sender"); pair.to_client.loss = 0.0; pair.drive(); assert!(grants(&mut pair) > lost, "stream credit was retransmitted"); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Available { dir: Dir::Uni })) - ); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + assert_eq!(open.wakes(), 1); // Try opening the second stream again, now that we've made room let s = pair @@ -1544,18 +1526,16 @@ fn stream_id_limit() { .open(Dir::Uni, Default::default()) .expect("didn't get stream id budget"); pair.client_send(client_ch, s).finish().unwrap(); + let accept = Counter::new(); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Uni) + .is_pending() + ); pair.drive(); // Make sure the server actually processes data on the newly-available stream - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Readable { id })) if id == s - ); + assert_eq!(accept.wakes(), 1); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); let mut recv = pair.server_recv(server_ch, s); let mut chunks = recv.read().unwrap(); @@ -1908,25 +1888,19 @@ fn key_update_simple() { const MSG1: &[u8] = b"hello1"; pair.client_send(client_ch, s).write(&mut &MSG1[..]).unwrap(); + let accept = Counter::new(); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Bi) + .is_pending() + ); pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Bi })) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Readable { id })) if id == s - ); + assert_eq!(accept.wakes(), 1); assert_matches!(pair.server_streams(server_ch).accept(Dir::Bi), Some(stream) if stream == s); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); - let mut recv = pair.server_recv(server_ch, s); - let mut chunks = recv.read().unwrap(); - assert_matches!( - chunks.next(usize::MAX), - Ok(Some(chunk)) if chunk.bytes == MSG1 - ); - let _ = chunks.finalize(); + let read = Counter::new(); + assert_matches!(poll_read(&mut pair, server_ch, s, &read), Poll::Ready(Ok(Some(data))) if data == MSG1); + assert!(poll_read(&mut pair, server_ch, s, &read).is_pending()); info!("initiating key update"); pair.client_conn_mut(client_ch).force_key_update(); @@ -1935,15 +1909,8 @@ fn key_update_simple() { pair.client_send(client_ch, s).write(&mut &MSG2[..]).unwrap(); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::Stream(StreamEvent::Readable { id })) if id == s); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); - let mut recv = pair.server_recv(server_ch, s); - let mut chunks = recv.read().unwrap(); - assert_matches!( - chunks.next(usize::MAX), - Ok(Some(chunk)) if chunk.bytes == MSG2 - ); - let _ = chunks.finalize(); + assert_eq!(read.wakes(), 1); + assert_matches!(poll_read(&mut pair, server_ch, s, &read), Poll::Ready(Ok(Some(data))) if data == MSG2); assert_eq!(pair.client_conn_mut(client_ch).stats().path.lost_packets, 0); assert_eq!(pair.server_conn_mut(server_ch).stats().path.lost_packets, 0); @@ -1979,10 +1946,6 @@ fn key_update_reordered() { pair.drive(); assert_eq!(pair.client_conn_mut(client_ch).stats().path.lost_packets, 0); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Bi })) - ); assert_matches!(pair.server_streams(server_ch).accept(Dir::Bi), Some(stream) if stream == s); let mut recv = pair.server_recv(server_ch, s); @@ -2005,8 +1968,8 @@ fn initial_retransmit() { pair.client.drive(pair.time, pair.server.addr); pair.client.outbound.clear(); // Drop initial pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); + assert!(has_handshake_data(pair.client_conn_mut(client_ch))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } /// Each side tells its endpoint once that the connection stopped being active, before it drains, @@ -2091,19 +2054,17 @@ fn instant_close_1() { pair.drive(); let server_ch = pair.server.assert_accept(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::LocallyClosed { code: VarInt(0), .. } - }) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::LocallyClosed { code: VarInt(0), .. }) ); + let server = pair.server_conn_mut(server_ch); + assert!(!has_handshake_data(server)); assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::ConnectionClosed(ConnectionClose { - error_code: TransportErrorCode::APPLICATION_ERROR, - .. - }), - }) + server.error(), + Some(ConnectionError::ConnectionClosed(ConnectionClose { + error_code: TransportErrorCode::APPLICATION_ERROR, + .. + })) ); } @@ -2122,21 +2083,19 @@ fn instant_close_2() { .close(pair.time, VarInt(42), String::new()); pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::LocallyClosed { code: VarInt(42), .. } - }) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::LocallyClosed { code: VarInt(42), .. }) ); let server_ch = pair.server.assert_accept(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeDataReady)); + let server = pair.server_conn_mut(server_ch); + assert!(has_handshake_data(server)); + assert!(!is_connected(server)); assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::ConnectionClosed(ConnectionClose { - error_code: TransportErrorCode::APPLICATION_ERROR, - .. - }), - }) + server.error(), + Some(ConnectionError::ConnectionClosed(ConnectionClose { + error_code: TransportErrorCode::APPLICATION_ERROR, + .. + })) ); } @@ -2157,13 +2116,11 @@ fn instant_server_close() { .close(pair.time, VarInt(42), String::new()); pair.drive(); assert_matches!( - pair.client_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::ConnectionClosed(ConnectionClose { - error_code: TransportErrorCode::APPLICATION_ERROR, - .. - }), - }) + pair.client_conn_mut(server_ch).error(), + Some(ConnectionError::ConnectionClosed(ConnectionClose { + error_code: TransportErrorCode::APPLICATION_ERROR, + .. + })) ); } @@ -2193,18 +2150,8 @@ fn idle_timeout() { } assert!(pair.time - start < Duration::from_millis(2 * IDLE_TIMEOUT)); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::TimedOut, - }) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::TimedOut, - }) - ); + assert_matches!(pair.client_conn_mut(client_ch).error(), Some(ConnectionError::TimedOut)); + assert_matches!(pair.server_conn_mut(server_ch).error(), Some(ConnectionError::TimedOut)); } #[test] @@ -2218,13 +2165,11 @@ fn unreachable_handshake_fails_with_no_viable_path() { let conn = pair.client_conn_mut(client_ch); assert!(conn.is_drained()); assert_matches!( - conn.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::TransportError(TransportError { - code: TransportErrorCode::NO_VIABLE_PATH, - .. - }), - }) + conn.error(), + Some(ConnectionError::TransportError(TransportError { + code: TransportErrorCode::NO_VIABLE_PATH, + .. + })) ); // Not even a CONNECTION_CLOSE goes out. pair.drive(); @@ -2237,20 +2182,20 @@ fn unreachable_is_loss_once_established() { let mut pair = Pair::default(); let (client_ch, server_ch) = pair.connect(); + let closed = Counter::new(); + assert!(pair.client_conn_mut(client_ch).poll_closed(&closed).is_pending()); pair.client_conn_mut(client_ch).handle_unreachable(); assert!(!pair.client_conn_mut(client_ch).is_closed()); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + assert_eq!(closed.wakes(), 0); + assert_matches!(pair.client_conn_mut(client_ch).error(), None); let s = pair .client_streams(client_ch) .open(Dir::Uni, Default::default()) .unwrap(); pair.client_send(client_ch, s).write(&mut &b"hello"[..]).unwrap(); pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); + assert_eq!(pair.server_streams(server_ch).accept(Dir::Uni), Some(s)); } #[test] @@ -2304,15 +2249,13 @@ fn connection_close_while_congestion_blocked() { .close(pair.time, VarInt(42), REASON.into()); // Step the simulation by hand so we can catch the exact moment the server hears about the - // close: check for the event after each packet exchange, before the clock jumps ahead + // close: check for the loss after each packet exchange, before the clock jumps ahead let mut result = None; loop { pair.drive_client(); pair.drive_server(); - while let Some(event) = pair.server_conn_mut(server_ch).poll() { - if let Event::ConnectionLost { reason } = event { - result = Some((reason, pair.time)); - } + if let Some(reason) = pair.server_conn_mut(server_ch).error() { + result = Some((reason.clone(), pair.time)); } if result.is_some() || !pair.step() { break; @@ -2336,8 +2279,8 @@ fn server_hs_retransmit() { assert!(!pair.client.inbound.is_empty()); // Initial + Handshakes pair.client.inbound.clear(); pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); + assert!(has_handshake_data(pair.client_conn_mut(client_ch))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } #[test] @@ -2359,7 +2302,10 @@ fn migration() { assert_ne!(pair.server_conn_mut(server_ch).total_recvd(), 0); pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + // The client's own path change is no news to it, and the connection survives it. + let client = pair.client_conn_mut(client_ch); + assert!(!iter::from_fn(|| client.poll()).any(|event| matches!(event, Event::PathUpdated))); + assert_matches!(pair.client_conn_mut(client_ch).error(), None); assert_eq!(pair.server_conn_mut(server_ch).remote_address(), pair.client.addr); // Assert that the client's response to the PATH_CHALLENGE was an IMMEDIATE_ACK, instead of a @@ -2546,12 +2492,15 @@ fn stop_opens_bidi() { .recv_stream(s) .stop(ERROR) .unwrap(); + let accept = Counter::new(); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Bi) + .is_pending() + ); pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Bi })) - ); + assert_eq!(accept.wakes(), 1); assert_eq!(pair.server_conn_mut(client_ch).streams().send_streams(), 0); assert_matches!(pair.server_streams(server_ch).accept(Dir::Bi), Some(stream) if stream == s); assert_eq!(pair.server_conn_mut(client_ch).streams().send_streams(), 1); @@ -2565,14 +2514,10 @@ fn stop_opens_bidi() { pair.server_send(server_ch, s).write(&mut &b"foo"[..]), Err(send::Error::Stopped(ERROR)) ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Stopped { - id: _, - error_code: ERROR - })) + assert_eq!( + peek(|waiter| pair.server_send(server_ch, s).poll_end(waiter)), + Poll::Ready(Ok(send::End::Stopped(ERROR))) ); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); } #[test] @@ -2589,11 +2534,14 @@ fn implicit_open() { .open(Dir::Uni, Default::default()) .unwrap(); pair.client_send(client_ch, s2).write(&mut &b"hello"[..]).unwrap(); - pair.drive(); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) + let accept = Counter::new(); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Uni) + .is_pending() ); + pair.drive(); + assert_eq!(accept.wakes(), 1); assert_eq!(pair.server_streams(server_ch).accept(Dir::Uni), Some(s1)); assert_eq!(pair.server_streams(server_ch).accept(Dir::Uni), Some(s2)); assert_eq!(pair.server_streams(server_ch).accept(Dir::Uni), None); @@ -2900,19 +2848,17 @@ fn finish_stream_flow_control_reordered() { pair.server.delay_outbound(); // Delay it pair.client_send(client_ch, s).finish().unwrap(); + let end = Counter::new(); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); pair.drive_client(); // Send FIN pair.server.drive(pair.time, pair.client.addr); // Acknowledge pair.server.finish_delay(); // Add flow control packets after pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Finished { id })) if id == s - ); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); @@ -3018,10 +2964,13 @@ fn stop_during_finish() { pair.server_recv(server_ch, s).stop(ERROR).unwrap(); pair.drive_server(); pair.client_send(client_ch, s).finish().unwrap(); + let end = Counter::new(); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); pair.drive_client(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Stopped { id, error_code: ERROR })) if id == s + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Stopped(ERROR))) ); } @@ -3254,7 +3203,8 @@ fn datagram_send_recv() { let _guard = subscribe(); let mut pair = Pair::default(); let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + let recv = Counter::new(); + assert!(pair.server_datagrams(server_ch).poll_recv(&recv).is_pending()); assert_matches!(pair.client_datagrams(client_ch).max_size(), Some(x) if x > 0); const DATA: &[u8] = b"whee"; @@ -3262,7 +3212,7 @@ fn datagram_send_recv() { .send(Bytes::from(DATA), Default::default()) .unwrap(); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::DatagramReceived)); + assert_eq!(recv.wakes(), 1); assert_eq!(pair.server_datagrams(server_ch).recv().unwrap(), DATA); assert_matches!(pair.server_datagrams(server_ch).recv(), None); } @@ -3344,7 +3294,8 @@ fn datagram_recv_buffer_overflow() { }; let mut pair = Pair::new(Default::default(), server); let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + let recv = Counter::new(); + assert!(pair.server_datagrams(server_ch).poll_recv(&recv).is_pending()); assert_eq!( pair.client_conn_mut(client_ch).datagrams().max_size(), Datagram::max_payload(WINDOW as u64).map(|n| n as usize) @@ -3363,7 +3314,7 @@ fn datagram_recv_buffer_overflow() { .send(Bytes::from(DATA3), Default::default()) .unwrap(); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::DatagramReceived)); + assert_eq!(recv.wakes(), 1); assert_eq!(pair.server_datagrams(server_ch).recv().unwrap(), DATA2); assert_eq!(pair.server_datagrams(server_ch).recv().unwrap(), DATA3); assert_matches!(pair.server_datagrams(server_ch).recv(), None); @@ -3394,7 +3345,8 @@ fn datagram_of_max_size_keeps_connection() { }; let mut pair = Pair::new(Default::default(), server); let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + let recv = Counter::new(); + assert!(pair.server_datagrams(server_ch).poll_recv(&recv).is_pending()); let max_size = pair.client_conn_mut(client_ch).datagrams().max_size().unwrap(); assert_eq!(max_size, Datagram::max_payload(WINDOW as u64).unwrap() as usize); @@ -3402,7 +3354,8 @@ fn datagram_of_max_size_keeps_connection() { .send(Bytes::from(vec![0xAB; max_size]), Default::default()) .unwrap(); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + // Dropped, so no receiver hears of it. + assert_eq!(recv.wakes(), 0); assert!(!pair.server_conn_mut(server_ch).is_closed()); const DATA: &[u8] = &[0xCD; 100]; @@ -3410,7 +3363,7 @@ fn datagram_of_max_size_keeps_connection() { .send(Bytes::from(DATA), Default::default()) .unwrap(); pair.drive(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::DatagramReceived)); + assert_eq!(recv.wakes(), 1); assert_eq!(pair.server_datagrams(server_ch).recv().unwrap(), DATA); } @@ -3477,8 +3430,7 @@ fn datagram_unsupported() { ..server_config() }; let mut pair = Pair::new(Default::default(), server); - let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + let (client_ch, _) = pair.connect(); assert_matches!(pair.client_datagrams(client_ch).max_size(), None); match pair.client_datagrams(client_ch).send(Bytes::new(), Default::default()) { @@ -3504,8 +3456,7 @@ fn datagram_zero_limit_unsupported() { ..server_config() }; let mut pair = Pair::new(Default::default(), server); - let (client_ch, server_ch) = pair.connect(); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); + let (client_ch, _) = pair.connect(); assert_matches!(pair.client_datagrams(client_ch).max_size(), None); assert_matches!( pair.client_datagrams(client_ch).send(Bytes::new(), Default::default()), @@ -3524,15 +3475,14 @@ fn large_initial() { let client_ch = pair.begin_connect(cfg); pair.drive(); let server_ch = pair.server.assert_accept(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.server_conn_mut(server_ch).poll(), Some(Event::Connected)); + let client = pair.client_conn_mut(client_ch); + assert!(has_handshake_data(client)); + assert!(is_connected(client)); + assert_established(pair.server_conn_mut(server_ch)); } #[test] -/// Ensure that we don't yield a finish event before the actual FIN is acked so the peer isn't left +/// Ensure that the end wait does not wake before the actual FIN is acked so the peer isn't left /// hanging fn finish_acked() { let _guard = subscribe(); @@ -3546,22 +3496,19 @@ fn finish_acked() { const MSG: &[u8] = b"hello"; pair.client_send(client_ch, s).write(&mut &MSG[..]).unwrap(); + let accept = Counter::new(); + assert!( + pair.server_streams(server_ch) + .poll_accept(&accept, Dir::Uni) + .is_pending() + ); info!("client sends data to server"); pair.drive_client(); // send data to server info!("server acknowledges data"); pair.drive_server(); // process data and send data ack // Receive data - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) - ); - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Readable { id })) if id == s - ); - assert_matches!(pair.server_conn_mut(server_ch).poll(), None); - + assert_eq!(accept.wakes(), 1); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); let mut recv = pair.server_recv(server_ch, s); @@ -3575,17 +3522,21 @@ fn finish_acked() { // Finish before receiving data ack pair.client_send(client_ch, s).finish().unwrap(); + let end = Counter::new(); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); // Send FIN, receive data ack info!("client receives ACK, sends FIN"); pair.drive_client(); // Check for premature finish from data ack - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + assert_eq!(end.wakes(), 0); + assert!(peek(|waiter| pair.client_send(client_ch, s).poll_end(waiter)).is_pending()); // Process FIN ack info!("server ACKs FIN"); pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Finished { id })) if id == s + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); let mut recv = pair.server_recv(server_ch, s); @@ -3595,7 +3546,7 @@ fn finish_acked() { } #[test] -/// Ensure that we don't yield a finish event while there's still unacknowledged data +/// Ensure that the end wait does not wake while there's still unacknowledged data fn finish_retransmit() { let _guard = subscribe(); let mut pair = Pair::default(); @@ -3613,23 +3564,22 @@ fn finish_retransmit() { // Send FIN pair.client_send(client_ch, s).finish().unwrap(); + let end = Counter::new(); + assert!(pair.client_send(client_ch, s).poll_end(&end).is_pending()); pair.drive_client(); // Process FIN pair.drive_server(); // Receive FIN ack, but no data ack pair.drive_client(); // Check for premature finish from FIN ack - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + assert_eq!(end.wakes(), 0); + assert!(peek(|waiter| pair.client_send(client_ch, s).poll_end(waiter)).is_pending()); // Recover pair.drive(); - assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::Stream(StreamEvent::Finished { id })) if id == s - ); - - assert_matches!( - pair.server_conn_mut(server_ch).poll(), - Some(Event::Stream(StreamEvent::Opened { dir: Dir::Uni })) + assert_eq!(end.wakes(), 1); + assert_eq!( + pair.client_send(client_ch, s).poll_end(&end), + Poll::Ready(Ok(send::End::Finished)) ); assert_matches!(pair.server_streams(server_ch).accept(Dir::Uni), Some(stream) if stream == s); @@ -3721,8 +3671,8 @@ fn handshake_anti_deadlock_probe() { pair.server.inbound.clear(); // Client sends an anti-deadlock probe, and the handshake completes as usual. pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); + assert!(has_handshake_data(pair.client_conn_mut(client_ch))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } /// The ack-eliciting frames the client has sent, which its probes carry @@ -4127,8 +4077,8 @@ fn handshake_outlasts_short_idle_timeout() { } pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); + assert!(has_handshake_data(pair.client_conn_mut(client_ch))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } /// A handshake idle timeout too long to add to an `Instant` never fires, instead of panicking. @@ -4212,8 +4162,8 @@ fn server_can_send_3_inital_packets() { assert_eq!(pair.client.inbound.len(), 3); pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(client_ch).poll(), Some(Event::Connected)); + assert!(has_handshake_data(pair.client_conn_mut(client_ch))); + assert!(is_connected(pair.client_conn_mut(client_ch))); } /// Generate a big fat certificate that can't fit inside the initial anti-amplification limit @@ -5028,10 +4978,8 @@ fn silently_drop_rejected_initials() { pair.time += Duration::from_secs(60); pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::HandshakeTimeout, - }) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::HandshakeTimeout) ); } @@ -5048,10 +4996,9 @@ fn reject_manually() { let client = pair.client.connections.get_mut(&client_ch).unwrap(); assert!(client.is_closed()); assert!(matches!( - client.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::ConnectionClosed(close) - }) if close.error_code == TransportErrorCode::CONNECTION_REFUSED + client.error(), + Some(ConnectionError::ConnectionClosed(close)) + if close.error_code == TransportErrorCode::CONNECTION_REFUSED )); } @@ -5081,13 +5028,17 @@ fn validate_then_reject_manually() { let client = pair.client.connections.get_mut(&client_ch).unwrap(); assert!(client.is_closed()); assert!(matches!( - client.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::ConnectionClosed(close) - }) if close.error_code == TransportErrorCode::CONNECTION_REFUSED + client.error(), + Some(ConnectionError::ConnectionClosed(close)) + if close.error_code == TransportErrorCode::CONNECTION_REFUSED )); pair.drive(); - assert_matches!(pair.client_conn_mut(client_ch).poll(), None); + // Draining keeps the refusal as the reason. + assert!(matches!( + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ConnectionClosed(close)) + if close.error_code == TransportErrorCode::CONNECTION_REFUSED + )); assert_eq!(pair.client.known_connections(), 0); assert_eq!(pair.client.known_cids(), 0); assert_eq!(pair.server.known_connections(), 0); @@ -5285,7 +5236,7 @@ fn min_mtu_limits_application_loss_probes() { let stream = client.streams().open(Dir::Uni, Default::default()).unwrap(); client.send_stream(stream).write(&mut &[42; 3000][..]).unwrap(); let mut buf = Vec::new(); - let transmit = client.poll_transmit(now, 1, &mut buf).unwrap(); + let transmit = client.transmit(now, 1, &mut buf).unwrap(); assert_eq!(transmit.size, usize::from(MIN_MTU)); assert_eq!(buf.len(), usize::from(MIN_MTU)); pair.drive(); @@ -5657,7 +5608,6 @@ fn path_changes_unblock_oversized_datagrams() { .send(data.clone(), Default::default()) .unwrap(); } - while pair.server_conn_mut(server_ch).poll().is_some() {} pair.mtu = 1200; if migrate { @@ -5815,23 +5765,19 @@ fn handshake_sequence() { let mut pair = Pair::new(Default::default(), server_config); let ch = pair.begin_connect(client_config()); + // Each step checks how far either side got, so the order shows across steps, though not + // within one. pair.step(); - assert_matches!(pair.client_conn_mut(ch).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (false, false, false)); let sh = pair.server.assert_accept(); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.server_conn_mut(sh).poll(), None); + assert_eq!(handshake_progress(pair.server_conn_mut(sh)), (true, false, false)); pair.step(); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::Connected)); - assert_matches!(pair.client_conn_mut(ch).poll(), None); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::Connected)); - assert_matches!(pair.server_conn_mut(sh).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (true, true, false)); + assert_eq!(handshake_progress(pair.server_conn_mut(sh)), (true, true, true)); pair.drive_client(); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.client_conn_mut(ch).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (true, true, true)); } #[test] @@ -5856,23 +5802,19 @@ fn handshake_confirmation_no_resumption_shortcut() { info!("resuming session"); let ch = pair.begin_connect(config); + // Each step checks how far either side got, so the order shows across steps, though not + // within one. pair.step(); - assert_matches!(pair.client_conn_mut(ch).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (false, false, false)); let sh = pair.server.assert_accept(); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.server_conn_mut(sh).poll(), None); + assert_eq!(handshake_progress(pair.server_conn_mut(sh)), (true, false, false)); pair.step(); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::Connected)); - assert_matches!(pair.client_conn_mut(ch).poll(), None); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.server_conn_mut(sh).poll(), Some(Event::Connected)); - assert_matches!(pair.server_conn_mut(sh).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (true, true, false)); + assert_eq!(handshake_progress(pair.server_conn_mut(sh)), (true, true, true)); pair.drive_client(); - assert_matches!(pair.client_conn_mut(ch).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(pair.client_conn_mut(ch).poll(), None); + assert_eq!(handshake_progress(pair.client_conn_mut(ch)), (true, true, true)); } /// A CONNECTION_CLOSE frame of type 0x1d must be rejected in an Initial packet @@ -5903,7 +5845,7 @@ fn application_close_in_initial_is_rejected() { // Grab the client's Initial packet so we can learn the connection IDs and version it chose. let mut buf = Vec::new(); let transmit = conn - .poll_transmit(now, 1, &mut buf) + .transmit(now, 1, &mut buf) .expect("client should send an Initial packet"); let initial = &buf[..transmit.size]; // Long header: flags(1) version(4) dcid_len(1) dcid scid_len(1) scid ... @@ -5960,13 +5902,11 @@ fn application_close_in_initial_is_rejected() { conn.handle_event(event); assert_matches!( - conn.poll(), - Some(Event::ConnectionLost { - reason: ConnectionError::TransportError(TransportError { - code: TransportErrorCode::PROTOCOL_VIOLATION, - .. - }) - }) + conn.error(), + Some(ConnectionError::TransportError(TransportError { + code: TransportErrorCode::PROTOCOL_VIOLATION, + .. + })) ); } @@ -6257,7 +6197,7 @@ fn send_quantum_bounds_the_gso_batch() { let mut buf = Vec::new(); let transmit = pair .client_conn_mut(client_ch) - .poll_transmit(now, MAX_DATAGRAMS, &mut buf) + .transmit(now, MAX_DATAGRAMS, &mut buf) .expect("a stream write must produce a transmit"); let datagrams = match transmit.segment_size { @@ -6316,7 +6256,7 @@ fn gso_train_fits_one_udp_send() { assert_eq!(conn.path_mtu(), MTU); let mut buf = Vec::new(); let mut full = false; - while let Some(transmit) = conn.poll_transmit(now, 64, &mut buf) { + while let Poll::Ready(transmit) = conn.poll_transmit(&kio::Waiter::noop(), now, 64, &mut buf) { assert!( transmit.size <= MAX_TRANSMIT_SIZE, "a {} byte train exceeds one IPv4 UDP send", @@ -6776,7 +6716,7 @@ fn partial_poll_reports_starvation() { log.lock().unwrap().clear(); let now = pair.time; let mut buf = Vec::new(); - assert!(pair.client_conn_mut(ch).poll_transmit(now, 10, &mut buf).is_some()); + assert!(pair.client_conn_mut(ch).transmit(now, 10, &mut buf).is_some()); assert!(matches!(log.lock().unwrap().last(), Some(PacketEvent::AppLimited { in_flight }) if *in_flight > 0)); } diff --git a/rs/moq-quic/src/tests/reset_stream_at.rs b/rs/moq-quic/src/tests/reset_stream_at.rs index 57ab253bb9e..e92d16c285d 100644 --- a/rs/moq-quic/src/tests/reset_stream_at.rs +++ b/rs/moq-quic/src/tests/reset_stream_at.rs @@ -430,9 +430,7 @@ fn unsupported_after_finish() { ); pair.drive(); assert_eq!(pair.client_streams(client_ch).send_streams(), 0); - while let Some(event) = pair.client_conn_mut(client_ch).poll() { - assert!(!matches!(event, Event::Stream(StreamEvent::Finished { .. }))); - } + assert_eq!(pair.client_send(client_ch, s).end(), Ok(Some(send::End::Reset(ERROR)))); } #[test] diff --git a/rs/moq-quic/src/tests/sim.rs b/rs/moq-quic/src/tests/sim.rs index 558d6076bc3..409884bf7ea 100644 --- a/rs/moq-quic/src/tests/sim.rs +++ b/rs/moq-quic/src/tests/sim.rs @@ -350,15 +350,13 @@ impl Peer { Ok(()) } - /// Drains events, failing if the connection was lost - fn poll(&self, pair: &mut Pair) -> Result<(), TestCaseError> { - while let Some(event) = self.conn(pair).poll() { - if let Event::ConnectionLost { reason } = event { - return Err(TestCaseError::fail(format!( - "{} lost the connection: {reason}", - self.side() - ))); - } + /// Fails if the connection was lost + fn check_alive(&self, pair: &mut Pair) -> Result<(), TestCaseError> { + if let Some(reason) = self.conn(pair).error() { + return Err(TestCaseError::fail(format!( + "{} lost the connection: {reason}", + self.side() + ))); } Ok(()) } @@ -469,8 +467,8 @@ fn run(case: &Case) -> Result { } next_read = pair.time + case.read_interval; } - client.poll(&mut pair)?; - server.poll(&mut pair)?; + client.check_alive(&mut pair)?; + server.check_alive(&mut pair)?; if client.done(&server.sends) && server.done(&client.sends) && extra.as_ref().is_none_or(|extra| extra.done()) { break; } @@ -518,12 +516,8 @@ fn run(case: &Case) -> Result { let now = pair.time; pair.client_conn_mut(client_ch).close(now, VarInt(0), String::new()); pair.drive(); - let reason = loop { - match pair.server_conn_mut(server_ch).poll() { - Some(Event::ConnectionLost { reason }) => break reason, - Some(_) => {} - None => return Err(TestCaseError::fail("server never saw the close")), - } + let Some(reason) = pair.server_conn_mut(server_ch).error().cloned() else { + return Err(TestCaseError::fail("server never saw the close")); }; prop_assert!( matches!(&reason, ConnectionError::ApplicationClosed(close) if close.error_code == VarInt(0)), diff --git a/rs/moq-quic/src/tests/token.rs b/rs/moq-quic/src/tests/token.rs index d3ae7451d36..5fe74839d1a 100644 --- a/rs/moq-quic/src/tests/token.rs +++ b/rs/moq-quic/src/tests/token.rs @@ -37,7 +37,7 @@ fn oversized_cached_initial_token() { .unwrap(); let mut buf = Vec::new(); assert_eq!( - conn.poll_transmit(now, 1, &mut buf).is_some(), + conn.transmit(now, 1, &mut buf).is_some(), fits, "token {token_len}, MTU {mtu}" ); @@ -46,11 +46,11 @@ fn oversized_cached_initial_token() { assert!(conn.stats().frame_tx.crypto > 0); } else { assert_matches!( - conn.poll(), - Some(Event::ConnectionLost { reason: ConnectionError::TransportError(err) }) + conn.error(), + Some(ConnectionError::TransportError(err)) if err.code == TransportErrorCode::PROTOCOL_VIOLATION ); - assert!(conn.poll_transmit(now, 1, &mut buf).is_none()); + assert!(conn.transmit(now, 1, &mut buf).is_none()); assert!(buf.is_empty()); } } @@ -81,7 +81,7 @@ fn oversized_retry_token() { ) .unwrap(); let mut buf = Vec::new(); - let _ = conn.poll_transmit(now, 1, &mut buf).unwrap(); + let _ = conn.transmit(now, 1, &mut buf).unwrap(); let Some(ProtectedHeader::Initial(initial)) = ProtectedHeader::decode(&mut Cursor::new(&buf), &FixedLengthConnectionIdParser::new(0), false).unwrap() else { @@ -117,17 +117,17 @@ fn oversized_retry_token() { let now = now + Duration::from_secs(1); let crypto_before = conn.stats().frame_tx.crypto; buf.clear(); - assert_eq!(conn.poll_transmit(now, 1, &mut buf).is_some(), fits); + assert_eq!(conn.transmit(now, 1, &mut buf).is_some(), fits); if fits { assert!(conn.stats().frame_tx.crypto > crypto_before); assert!(buf.len() <= 1200); } else { assert_matches!( - conn.poll(), - Some(Event::ConnectionLost { reason: ConnectionError::TransportError(err) }) + conn.error(), + Some(ConnectionError::TransportError(err)) if err.code == TransportErrorCode::PROTOCOL_VIOLATION ); - assert!(conn.poll_transmit(now, 1, &mut buf).is_none()); + assert!(conn.transmit(now, 1, &mut buf).is_none()); assert!(buf.is_empty()); } } @@ -174,8 +174,8 @@ async fn retry_token_expired() { pair.drive(); assert_matches!( - pair.client_conn_mut(client_ch).poll(), - Some(Event::ConnectionLost { reason: ConnectionError::ConnectionClosed(err) }) + pair.client_conn_mut(client_ch).error(), + Some(ConnectionError::ConnectionClosed(err)) if err.error_code == TransportErrorCode::INVALID_TOKEN ); diff --git a/rs/moq-quic/src/tests/util.rs b/rs/moq-quic/src/tests/util.rs index 803780685e4..a9d55b8d3f2 100644 --- a/rs/moq-quic/src/tests/util.rs +++ b/rs/moq-quic/src/tests/util.rs @@ -7,6 +7,7 @@ use std::{ net::{Ipv6Addr, SocketAddr, UdpSocket}, str, sync::{Arc, LazyLock, Mutex}, + task::Poll, }; use assert_matches::assert_matches; @@ -250,13 +251,10 @@ impl Pair { client_ch } + /// Asserts both sides finished the handshake pub(super) fn finish_connect(&mut self, client_ch: ConnectionHandle, server_ch: ConnectionHandle) { - assert_matches!(self.client_conn_mut(client_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(self.client_conn_mut(client_ch).poll(), Some(Event::Connected)); - assert_matches!(self.server_conn_mut(server_ch).poll(), Some(Event::HandshakeDataReady)); - assert_matches!(self.server_conn_mut(server_ch).poll(), Some(Event::HandshakeConfirmed)); - assert_matches!(self.server_conn_mut(server_ch).poll(), Some(Event::Connected)); - assert_matches!(self.client_conn_mut(client_ch).poll(), Some(Event::HandshakeConfirmed)); + assert_established(self.client_conn_mut(client_ch)); + assert_established(self.server_conn_mut(server_ch)); } pub(super) fn client_conn_mut(&mut self, ch: ConnectionHandle) -> &mut Connection { @@ -471,7 +469,7 @@ impl TestEndpoint { while let Some(event) = conn.poll_endpoint_events() { endpoint_events.push((ch, event)); } - while let Some(transmit) = conn.poll_transmit(now, MAX_DATAGRAMS, &mut buf) { + while let Some(transmit) = conn.transmit(now, MAX_DATAGRAMS, &mut buf) { let size = transmit.size; self.outbound.extend(split_transmit(transmit, &buf[..size])); buf.clear(); @@ -621,6 +619,59 @@ impl ::std::ops::DerefMut for TestEndpoint { } } +/// Poll the server's next chunk of `s` +pub(super) fn poll_read( + pair: &mut Pair, + ch: ConnectionHandle, + s: StreamId, + waiter: &kio::Waiter, +) -> Poll, recv::Error>> { + let mut recv = pair.server_recv(ch, s); + let mut chunks = recv.read().unwrap(); + let poll = chunks + .poll_next(waiter, usize::MAX) + .map(|r| r.map(|chunk| chunk.map(|c| c.bytes))); + let _ = chunks.finalize(); + poll +} + +/// Polls a wait with a waiter nothing wakes, to read its answer as it stands +pub(super) fn peek(poll: impl FnOnce(&kio::Waiter) -> Poll) -> Poll { + poll(&kio::Waiter::noop()) +} + +/// Asserts the connection has its handshake data, is connected and confirmed, and is still alive +/// +/// Waits count wakes rather than order them, so this checks where the handshake ended, not the +/// order its steps completed in. +pub(super) fn assert_established(conn: &mut Connection) { + assert!(has_handshake_data(conn)); + assert!(is_connected(conn)); + assert!(is_confirmed(conn)); + assert_matches!(conn.error(), None); +} + +/// How far the connection's handshake got: whether it has its handshake data, is connected, and +/// is confirmed +pub(super) fn handshake_progress(conn: &mut Connection) -> (bool, bool, bool) { + (has_handshake_data(conn), is_connected(conn), is_confirmed(conn)) +} + +/// Whether the connection's handshake data is available +pub(super) fn has_handshake_data(conn: &mut Connection) -> bool { + peek(|waiter| conn.poll_handshake_data(waiter)) == Poll::Ready(Ok(())) +} + +/// Whether the connection completed its handshake +pub(super) fn is_connected(conn: &mut Connection) -> bool { + peek(|waiter| conn.poll_connected(waiter)) == Poll::Ready(Ok(())) +} + +/// Whether the connection's handshake was confirmed +pub(super) fn is_confirmed(conn: &mut Connection) -> bool { + peek(|waiter| conn.poll_handshake_confirmed(waiter)) == Poll::Ready(Ok(())) +} + /// Logs the test's own events, and the stack's per-packet trace when `RUST_LOG` asks for it. /// /// Formatting every trace event costs more than the simulation it describes, and the runs are diff --git a/rs/moq-quic/src/tests/waits.rs b/rs/moq-quic/src/tests/waits.rs index a8550c318d7..d4b052a871b 100644 --- a/rs/moq-quic/src/tests/waits.rs +++ b/rs/moq-quic/src/tests/waits.rs @@ -28,22 +28,6 @@ fn open_uni(pair: &mut Pair, client_ch: ConnectionHandle, server_ch: ConnectionH s } -/// Poll the server's next chunk of `s` -fn poll_read( - pair: &mut Pair, - ch: ConnectionHandle, - s: StreamId, - waiter: &kio::Waiter, -) -> Poll, recv::Error>> { - let mut recv = pair.server_recv(ch, s); - let mut chunks = recv.read().unwrap(); - let poll = chunks - .poll_next(waiter, usize::MAX) - .map(|r| r.map(|chunk| chunk.map(|c| c.bytes))); - let _ = chunks.finalize(); - poll -} - #[test] fn read_wakes_every_waiter() { let _guard = subscribe(); @@ -368,6 +352,98 @@ fn handshake_waits() { assert_eq!(conn.poll_connected(&connected), Poll::Ready(Ok(()))); } +#[test] +fn handshake_data_waits() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let client_ch = pair.begin_connect(client_config()); + + let waiter = Counter::new(); + assert!( + pair.client_conn_mut(client_ch) + .poll_handshake_data(&waiter) + .is_pending() + ); + pair.drive(); + assert_eq!(waiter.wakes(), 1); + let now = pair.time; + let conn = pair.client_conn_mut(client_ch); + assert_eq!(conn.poll_handshake_data(&waiter), Poll::Ready(Ok(()))); + assert!(conn.handshake_data().is_some()); + + // Still available once lost: the parameters did arrive. + conn.close(now, VarInt(0), String::new()); + assert_eq!(conn.poll_handshake_data(&waiter), Poll::Ready(Ok(()))); +} + +/// A connection lost before its handshake parameters arrived fails the wait with the reason. +#[test] +fn handshake_data_fails_once_lost_first() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let client_ch = pair.begin_connect(client_config()); + + let waiter = Counter::new(); + let now = pair.time; + let conn = pair.client_conn_mut(client_ch); + assert!(conn.poll_handshake_data(&waiter).is_pending()); + conn.close(now, VarInt(7), String::new()); + assert_eq!(waiter.wakes(), 1); + let reason = assert_matches!(conn.poll_handshake_data(&waiter), Poll::Ready(Err(reason)) => reason); + assert_matches!(&reason, ConnectionError::LocallyClosed { code: VarInt(7), .. }); + assert_eq!(conn.error(), Some(&reason)); +} + +#[test] +fn received_reset_waits_for_the_reset() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let (client_ch, server_ch) = pair.connect(); + let s = open_uni(&mut pair, client_ch, server_ch, b"a"); + + let waiter = Counter::new(); + assert!(pair.server_recv(server_ch, s).poll_received_reset(&waiter).is_pending()); + + // Data wakes it too, and the answer is still no reset. + pair.client_send(client_ch, s).write(&mut &b"b"[..]).unwrap(); + pair.drive(); + assert_eq!(waiter.wakes(), 1); + assert!(pair.server_recv(server_ch, s).poll_received_reset(&waiter).is_pending()); + + pair.client_send(client_ch, s).reset(VarInt(5)).unwrap(); + pair.drive(); + assert_eq!(waiter.wakes(), 2); + assert_eq!( + pair.server_recv(server_ch, s).poll_received_reset(&waiter), + Poll::Ready(Ok(VarInt(5))) + ); + // Observing the reset discards the stream. + assert_eq!( + pair.server_recv(server_ch, s).poll_received_reset(&waiter), + Poll::Ready(Err(recv::Error::Closed)) + ); +} + +/// A stream the connection's loss ends without a reset fails the wait. +#[test] +fn received_reset_fails_once_lost() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let (client_ch, server_ch) = pair.connect(); + let s = open_uni(&mut pair, client_ch, server_ch, b"a"); + + let waiter = Counter::new(); + assert!(pair.server_recv(server_ch, s).poll_received_reset(&waiter).is_pending()); + let now = pair.time; + pair.client_conn_mut(client_ch).close(now, VarInt(0), String::new()); + pair.drive(); + assert_eq!(waiter.wakes(), 1); + assert_eq!( + pair.server_recv(server_ch, s).poll_received_reset(&waiter), + Poll::Ready(Err(recv::Error::ConnectionLost)) + ); +} + /// What arrived before the loss stays available, then every wait fails with the reason. #[test] fn lost_drains_then_fails() { @@ -469,3 +545,104 @@ fn stop_wakes_a_parked_reader() { Err(recv::Error::Closed) )); } + +/// Poll the client's transmit until it waits, parking `driver` on it +fn park_transmit(pair: &mut Pair, ch: ConnectionHandle, driver: &kio::Waiter) { + let now = pair.time; + let mut buf = Vec::new(); + let conn = pair.client_conn_mut(ch); + while conn.poll_transmit(driver, now, 1, &mut buf).is_ready() { + buf.clear(); + } +} + +/// What the application queues wakes the driver parked on `poll_transmit`, which then finds it +/// to send; nothing else does. +#[test] +fn transmit_waits_for_the_application() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let (client_ch, server_ch) = pair.connect(); + let s = open_uni(&mut pair, client_ch, server_ch, b"a"); + + let driver = Counter::new(); + park_transmit(&mut pair, client_ch, &driver); + // Reading what is already sent queues nothing. + assert_eq!(pair.client_send(client_ch, s).end(), Ok(None)); + assert_eq!(driver.wakes(), 0); + + pair.client_send(client_ch, s).write(&mut &b"b"[..]).unwrap(); + assert_eq!(driver.wakes(), 1); + park_transmit(&mut pair, client_ch, &driver); + pair.client_send(client_ch, s).finish().unwrap(); + assert_eq!(driver.wakes(), 2); + park_transmit(&mut pair, client_ch, &driver); + pair.client_datagrams(client_ch) + .send(&b"hi"[..], Default::default()) + .unwrap(); + assert_eq!(driver.wakes(), 3); + park_transmit(&mut pair, client_ch, &driver); + let now = pair.time; + pair.client_conn_mut(client_ch).close(now, VarInt(0), String::new()); + assert_eq!(driver.wakes(), 4); + // The close is what it finds to send. + let mut buf = Vec::new(); + assert!( + pair.client_conn_mut(client_ch) + .poll_transmit(&driver, now, 1, &mut buf) + .is_ready() + ); +} + +/// A reset, a stop, and a refused open each queue a frame, and wake the driver for it. +#[test] +fn transmit_hears_resets_stops_and_refused_opens() { + let _guard = subscribe(); + let mut pair = pair_with(Limits { + max_streams: NonZeroU64::new(1), + ..Limits::default() + }); + let (client_ch, server_ch) = pair.connect(); + let s = open_uni(&mut pair, client_ch, server_ch, b"a"); + + let driver = Counter::new(); + park_transmit(&mut pair, client_ch, &driver); + // The peer allows one stream, which is open. + assert_eq!(pair.client_streams(client_ch).open(Dir::Uni, Default::default()), None); + assert_eq!(driver.wakes(), 1, "STREAMS_BLOCKED"); + park_transmit(&mut pair, client_ch, &driver); + pair.client_send(client_ch, s).reset(VarInt(7)).unwrap(); + assert_eq!(driver.wakes(), 2, "RESET_STREAM"); + + let t = pair + .server_streams(server_ch) + .open(Dir::Bi, Default::default()) + .unwrap(); + pair.server_send(server_ch, t).write(&mut &b"x"[..]).unwrap(); + pair.drive(); + assert_eq!(pair.client_streams(client_ch).accept(Dir::Bi), Some(t)); + park_transmit(&mut pair, client_ch, &driver); + pair.client_conn_mut(client_ch).recv_stream(t).stop(VarInt(9)).unwrap(); + assert_eq!(driver.wakes(), 3, "STOP_SENDING"); +} + +/// A local address change queues a PING, and wakes the driver for it. +#[test] +fn transmit_hears_a_local_address_change() { + let _guard = subscribe(); + let mut pair = Pair::default(); + let (client_ch, _) = pair.connect(); + + let driver = Counter::new(); + park_transmit(&mut pair, client_ch, &driver); + pair.client_conn_mut(client_ch).local_address_changed(); + assert_eq!(driver.wakes(), 1, "PING"); + // The PING is what it finds to send. + let now = pair.time; + let mut buf = Vec::new(); + assert!( + pair.client_conn_mut(client_ch) + .poll_transmit(&driver, now, 1, &mut buf) + .is_ready() + ); +} diff --git a/rs/moq-tokio/Cargo.toml b/rs/moq-tokio/Cargo.toml index 1591a40fdbb..17713cf8a09 100644 --- a/rs/moq-tokio/Cargo.toml +++ b/rs/moq-tokio/Cargo.toml @@ -136,6 +136,7 @@ libc = { workspace = true } anyhow = "1" bytes = "1" chrono = "0.4" +criterion = { package = "bench-harness", path = "../bench-harness" } # Use the channel type behind `mdns_sd::Receiver` so the mDNS readiness gate can # be driven with synthetic daemon events. flume = "0.12" @@ -178,5 +179,12 @@ required-features = ["net"] name = "integration" path = "tests/integration/main.rs" required-features = ["net"] + +# Stream fanout through the driver on localhost, swept over streams per connection. +[[bench]] +name = "fanout" +harness = false +required-features = ["driver", "aws-lc-rs"] + [lints] workspace = true diff --git a/rs/moq-tokio/benches/fanout.rs b/rs/moq-tokio/benches/fanout.rs new file mode 100644 index 00000000000..8a6d091b3f2 --- /dev/null +++ b/rs/moq-tokio/benches/fanout.rs @@ -0,0 +1,154 @@ +//! Stream fanout through the tokio QUIC driver: one connection, many short streams at once. +//! +//! Each iteration the client opens N unidirectional streams on a long-lived localhost +//! connection, writes a small message on each and finishes it, while the server accepts and +//! reads every stream to the end. Reported per stream, so a cost that grows with the number of +//! streams in flight (a scan over every stream per event, say) shows as a slope across N. + +use std::net::{Ipv4Addr, SocketAddr, UdpSocket}; +use std::num::NonZeroU64; +use std::sync::Arc; +use std::time::Duration; + +use criterion::{BenchmarkId, Criterion, Throughput, criterion_group, criterion_main}; +use futures::{StreamExt as _, stream::FuturesUnordered}; +use moq_tokio::quic::driver::{ + Limits, client, + connection::Connection, + endpoint::{self, Endpoint}, + server, transport, +}; +use rustls::{ + RootCertStore, + pki_types::{PrivateKeyDer, PrivatePkcs8KeyDer}, +}; +use tokio::runtime::Runtime; + +/// Streams in flight per iteration. +const STREAMS: [usize; 4] = [1, 16, 256, 1024]; +/// Bytes written on each stream. +const MESSAGE: usize = 1024; + +/// A client connected to a server over localhost, each on its own endpoint. +struct Pair { + runtime: Runtime, + client: Connection, + server: Connection, + // Dropping an endpoint closes its connections. + _endpoints: [Endpoint; 2], +} + +impl Pair { + fn new() -> Self { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .expect("runtime"); + + // Credit for every stream and byte of the largest batch, so an iteration measures + // stream handling rather than waiting on the peer to extend flow control. + let mut limits = Limits::default(); + limits.max_streams = NonZeroU64::new(4 * STREAMS[STREAMS.len() - 1] as u64); + limits.receive_window = NonZeroU64::new(64 * 1024 * 1024); + let mut transport = transport::Config::default(); + transport.limits = limits; + + let cert = rcgen::generate_simple_self_signed(vec!["localhost".into()]).expect("certificate"); + let key = PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(cert.signing_key.serialize_der())); + let mut server_config = + server::Config::with_single_cert(vec![cert.cert.der().clone()], key).expect("server config"); + server_config.transport = transport.clone(); + let mut roots = RootCertStore::empty(); + roots.add(cert.cert.der().clone()).expect("root"); + let mut client_config = client::Config::with_root_certificates(Arc::new(roots)).expect("client config"); + client_config.transport = transport; + + let localhost = SocketAddr::from((Ipv4Addr::LOCALHOST, 0)); + let (client, server, endpoints) = runtime.block_on(async { + let server_endpoint = Endpoint::new( + endpoint::Config::default().with_server(server_config), + UdpSocket::bind(localhost).expect("bind"), + ) + .expect("server endpoint"); + let client_endpoint = Endpoint::new(endpoint::Config::default(), UdpSocket::bind(localhost).expect("bind")) + .expect("client endpoint"); + let connecting = client_endpoint + .connect(&client_config, server_endpoint.local_addr(), "localhost") + .expect("connect"); + let accepting = async { + let incoming = server_endpoint.accept().await.expect("incoming"); + incoming.accept().expect("accept").await.expect("server handshake") + }; + let (client, server) = tokio::join!(connecting, accepting); + ( + client.expect("client handshake"), + server, + [client_endpoint, server_endpoint], + ) + }); + + Self { + runtime, + client, + server, + _endpoints: endpoints, + } + } + + /// Send `streams` messages, each on its own stream, and wait until the server read them all. + fn fanout(&self, streams: usize, message: &[u8]) { + self.runtime.block_on(async { + let send = async { + let mut writes: FuturesUnordered<_> = (0..streams) + .map(|_| async { + let mut stream = self.client.open_uni(Default::default()).await.expect("open"); + stream.write_all(message).await.expect("write"); + // The server reading to the end is the iteration's completion, so the + // acknowledgement `Finished::stopped` waits for is left unawaited. + drop(stream.finish()); + }) + .collect(); + while writes.next().await.is_some() {} + }; + + let recv = async { + let mut reads = FuturesUnordered::new(); + let (mut accepted, mut read) = (0, 0); + while read < streams { + tokio::select! { + stream = self.server.accept_uni(), if accepted < streams => { + accepted += 1; + let mut stream = stream.expect("accept"); + reads.push(async move { stream.read_to_end(usize::MAX).await.expect("read").len() }); + } + Some(len) = reads.next(), if !reads.is_empty() => { + assert_eq!(len, message.len()); + read += 1; + } + } + } + }; + + tokio::join!(send, recv); + }); + } +} + +fn bench_fanout(c: &mut Criterion) { + let pair = Pair::new(); + let message = vec![0x5a; MESSAGE]; + let mut group = c.benchmark_group("fanout"); + group.sample_size(20); + group.warm_up_time(Duration::from_secs(1)); + group.measurement_time(Duration::from_secs(5)); + for streams in STREAMS { + group.throughput(Throughput::Elements(streams as u64)); + group.bench_with_input(BenchmarkId::new("uni_1KiB", streams), &streams, |b, &streams| { + b.iter(|| pair.fanout(streams, &message)); + }); + } + group.finish(); +} + +criterion_group!(benches, bench_fanout); +criterion_main!(benches); diff --git a/rs/moq-tokio/src/quic/driver/connection.rs b/rs/moq-tokio/src/quic/driver/connection.rs index f8d53bc5be5..785368ee0b2 100644 --- a/rs/moq-tokio/src/quic/driver/connection.rs +++ b/rs/moq-tokio/src/quic/driver/connection.rs @@ -7,20 +7,14 @@ use std::{ net::SocketAddr, pin::Pin, sync::{ - Arc, OnceLock, + Arc, atomic::{AtomicUsize, Ordering}, }, - task::{Context, Poll, Waker, ready}, + task::{Context, Poll, ready}, }; use bytes::{Buf, Bytes}; -use pin_project_lite::pin_project; -use rustc_hash::FxHashMap; -use tokio::sync::{ - Notify, - futures::{Notified, OwnedNotified}, - mpsc, oneshot, -}; +use tokio::sync::mpsc; use tracing::{Instrument, Span, debug_span}; use crate::quic::driver::{ @@ -29,7 +23,7 @@ use crate::quic::driver::{ use moq_congestion::Controller; pub use moq_quic::connection::{Error, Stats}; use moq_quic::{ - EndpointEvent, StreamEvent, TransportError, TransportErrorCode, crypto, + EndpointEvent, TransportError, TransportErrorCode, crypto, endpoint::ConnectionHandle, rustls::pki_types::CertificateDer, stream::{Dir, Id as StreamId}, @@ -45,8 +39,8 @@ use moq_quic::{ #[derive(Debug)] pub struct Connecting { conn: Option, - connected: Pin>, - handshake_data_ready: Option>, + /// Holds the waiter of [`Future::poll`] across polls + park: kio::Park, } impl Connecting { @@ -58,7 +52,6 @@ impl Connecting { slot: Arc, runtime: &crate::runtime::Spawner, ) -> Self { - let (on_handshake_data_send, on_handshake_data_recv) = oneshot::channel(); let shared = Shared { #[cfg(feature = "net")] probe: conn.congestion_state().probes().then(moq_net::transport::Probe::new), @@ -66,17 +59,9 @@ impl Connecting { }; let conn = ConnectionRef(Arc::new(ConnectionInner { - state: Mutex::new(State::new( - conn, - handle, - endpoint_events, - conn_events, - on_handshake_data_send, - slot, - )), + state: Mutex::new(State::new(conn, handle, endpoint_events, conn_events, slot)), shared, })); - let connected = Box::pin(conn.shared.connected.clone().notified_owned()); let driver = ConnectionDriver::new(conn.clone()); runtime.spawn( @@ -90,8 +75,7 @@ impl Connecting { Self { conn: Some(conn), - connected, - handshake_data_ready: Some(on_handshake_data_recv), + park: kio::Park::default(), } } @@ -101,21 +85,16 @@ impl Connecting { /// /// This operation is cancel-safe. pub async fn handshake_data(&mut self) -> Result { - // Taking &mut self allows us to use a single oneshot channel rather than dealing with - // potentially many tasks waiting on the same event. It's a bit of a hack, but keeps things - // simple. - if let Some(x) = self.handshake_data_ready.as_mut() { - let _ = x.await; - // Once the data is ready, apply state changes. This prevents a panic due to - // inconsistent state when retrying a call to `handshake_data`. - self.handshake_data_ready = None; - } let conn = self.conn.as_ref().unwrap(); - let inner = conn.state.lock(); - inner - .inner - .handshake_data() - .ok_or_else(|| inner.error.clone().expect("spurious handshake data ready notification")) + kio::wait(|waiter| { + let mut state = conn.state.lock(); + if let Err(err) = ready!(state.inner.poll_handshake_data(waiter)) { + return Poll::Ready(Err(state.lost_as(err))); + } + let data = state.inner.handshake_data(); + Poll::Ready(data.ok_or_else(|| state.lost())) + }) + .await } /// The peer's UDP address @@ -126,7 +105,9 @@ impl Connecting { conn_ref.state.lock().inner.remote_address() } - /// Panic while holding the state, as a driver bug would, then wake the driver into it. + /// Panic while holding the state, as a driver bug would. + /// + /// The driver reaches it on its next poll: its first, as it has not run yet, or its timer. #[cfg(test)] pub(super) fn poison(&self) { let conn = self.conn.as_ref().expect("used after yielding Ready"); @@ -134,26 +115,20 @@ impl Connecting { let _guard = conn.state.lock(); panic!("injected"); })); - conn.state.poisoned().expect("the panic poisoned the state").wake(); } } impl Future for Connecting { type Output = Result; fn poll(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - Pin::new(&mut self.connected).poll(cx).map(|_| { - let conn = self.conn.take().unwrap(); - let inner = conn.state.lock(); - if inner.connected { - drop(inner); - Ok(Connection(conn)) - } else { - Err(inner - .error - .clone() - .expect("connected signaled without connection success or error")) - } - }) + let this = &mut *self; + let waiter = this.park.hold(cx); + let conn = this.conn.as_ref().expect("used after yielding Ready"); + let mut state = conn.state.lock(); + let connected = ready!(state.inner.poll_connected(waiter)).map_err(|err| state.lost_as(err)); + drop(state); + let conn = this.conn.take().unwrap(); + Poll::Ready(connected.map(|()| Connection(conn))) } } @@ -172,6 +147,8 @@ impl Future for Connecting { struct ConnectionDriver { conn: ConnectionRef, span: Span, + /// Holds the waiter `poll_transmit` parks on, which the application's calls wake + park: kio::Park, } impl ConnectionDriver { @@ -181,7 +158,11 @@ impl ConnectionDriver { debug_span!("drive", id = conn.handle.0) }; - Self { conn, span } + Self { + conn, + span, + park: kio::Park::default(), + } } } @@ -189,52 +170,55 @@ impl Future for ConnectionDriver { type Output = Result<(), io::Error>; fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll { - let conn = &mut *self.conn.state.lock(); - let _guard = self.span.enter(); + let this = self.get_mut(); + let waiter = this.park.hold(cx); + let conn = &mut *this.conn.state.lock(); + let _guard = this.span.enter(); - if let Err(e) = conn.process_conn_events(&self.conn.shared, cx) { - conn.terminate(e, &self.conn.shared); + if let Err(e) = conn.process_conn_events(cx) { + conn.terminate(e); return Poll::Ready(Ok(())); } // A new target waits for the next poll: probing only resends data in flight, so it has // nothing to send until a write or an ACK polls anyway. #[cfg(feature = "net")] - if let Some(probe) = &self.conn.shared.probe { + if let Some(probe) = &this.conn.shared.probe { conn.inner.set_probe_target(probe.target().as_bps() / 8); } - let mut keep_going = conn.drive_transmit(cx).inspect_err(|_| { + let mut keep_going = conn.drive_transmit(waiter, cx).inspect_err(|_| { // Transmit failed, so close the connection and clean state before returning the error. if !conn.inner.is_closed() { - conn.implicit_close(&self.conn.shared); + conn.implicit_close(); } })?; // If a timer expires, there might be more to transmit. When we transmit something, we // might need to reset a timer. Hence, we must loop until neither happens. keep_going |= conn.drive_timer(cx); conn.forward_endpoint_events(); - conn.forward_app_events(&self.conn.shared); + conn.drain_events(); // Every datagram sent or received passes through this poll. #[cfg(feature = "net")] { let stats = conn.inner.stats(); - self.conn.shared.wire.set_sent_bytes(stats.udp_tx.bytes); - self.conn.shared.wire.set_received_bytes(stats.udp_rx.bytes); + this.conn.shared.wire.set_sent_bytes(stats.udp_tx.bytes); + this.conn.shared.wire.set_received_bytes(stats.udp_rx.bytes); } if !conn.inner.is_drained() { + // A local close still reaches a driver whose socket holds output back, which skips + // `poll_transmit` and so parks nothing there, to re-arm the timer for its closing + // period. Waiting on the close consumes no transmit. + let _ = conn.inner.poll_closed(waiter); + // Otherwise `poll_transmit` parked the waiter for the application's next call, and the + // ingress channel, the timer, and the socket hold the context. if keep_going { // If the connection hasn't processed all tasks, schedule it again cx.waker().wake_by_ref(); - } else { - match conn.driver.as_mut() { - Some(old_waker) => old_waker.clone_from(cx.waker()), - None => conn.driver = Some(cx.waker().clone()), - }; } return Poll::Pending; } - if conn.error.is_none() { + if !conn.is_lost() { unreachable!("drained connections always have an error"); } Poll::Ready(Ok(())) @@ -250,7 +234,7 @@ impl Drop for ConnectionDriver { TransportErrorCode::INTERNAL_ERROR, "connection driver panicked".to_string(), ); - conn.terminate(Error::TransportError(reason), &self.conn.shared); + conn.terminate(Error::TransportError(reason)); } } } @@ -282,15 +266,9 @@ impl Connection { /// /// # Panics /// - when `options.group` belongs to another connection - pub fn open_uni<'a>( - &'a self, - options: send::Options<'a>, - ) -> impl Future> + 'a { - OpenUni { - conn: &self.0, - options, - notify: self.0.shared.stream_budget_available[Dir::Uni as usize].notified(), - } + pub async fn open_uni(&self, options: send::Options<'_>) -> Result { + let (conn, id) = kio::wait(|waiter| poll_open(waiter, &self.0, options, Dir::Uni)).await?; + Ok(send::Stream::new(conn, id)) } /// Open a send group at `priority`, for streams and datagrams to join with [`send::Options`] @@ -310,23 +288,17 @@ impl Connection { /// /// # Panics /// - when `options.group` belongs to another connection - pub fn open_bi<'a>( - &'a self, - options: send::Options<'a>, - ) -> impl Future> + 'a { - OpenBi { - conn: &self.0, - options, - notify: self.0.shared.stream_budget_available[Dir::Bi as usize].notified(), - } + pub async fn open_bi(&self, options: send::Options<'_>) -> Result<(send::Stream, recv::Stream), Error> { + let (conn, id) = kio::wait(|waiter| poll_open(waiter, &self.0, options, Dir::Bi)).await?; + Ok((send::Stream::new(conn.clone(), id), recv::Stream::new(conn, id))) } /// Open a request stream using only the credit already granted by the peer. #[cfg(feature = "net")] pub(crate) fn open_request(&self) -> Result, Error> { let mut state = self.0.state.lock(); - if let Some(err) = &state.error { - return Err(err.clone()); + if let Some(err) = state.error() { + return Err(err); } let Some(id) = state.inner.streams().open(Dir::Bi, Default::default()) else { return Ok(None); @@ -343,11 +315,9 @@ impl Connection { /// # Cancel safety /// /// This method is cancellation safe. If this does not resolve, no streams were consumed. - pub fn accept_uni(&self) -> impl Future> + '_ { - AcceptUni { - conn: &self.0, - notify: self.0.shared.stream_incoming[Dir::Uni as usize].notified(), - } + pub async fn accept_uni(&self) -> Result { + let (conn, id) = kio::wait(|waiter| poll_accept(waiter, &self.0, Dir::Uni)).await?; + Ok(recv::Stream::new(conn, id)) } /// Accept the next incoming bidirectional stream @@ -362,11 +332,9 @@ impl Connection { /// # Cancel safety /// /// This method is cancellation safe. If this does not resolve, no streams were consumed. - pub fn accept_bi(&self) -> impl Future> + '_ { - AcceptBi { - conn: &self.0, - notify: self.0.shared.stream_incoming[Dir::Bi as usize].notified(), - } + pub async fn accept_bi(&self) -> Result<(send::Stream, recv::Stream), Error> { + let (conn, id) = kio::wait(|waiter| poll_accept(waiter, &self.0, Dir::Bi)).await?; + Ok((send::Stream::new(conn.clone(), id), recv::Stream::new(conn, id))) } /// Receive an application datagram @@ -375,10 +343,13 @@ impl Connection { /// /// This method is cancellation safe. If this does not resolve, no datagrams were consumed. pub fn read_datagram(&self) -> impl Future> + '_ { - ReadDatagram { - conn: &self.0, - notify: self.0.shared.datagram_received.notified(), - } + kio::wait(|waiter| { + let mut state = self.0.state.lock(); + // moq-quic yields the datagrams received before checking for the connection error, + // so they, which are necessarily finite, drain from a closed connection. + let datagram = ready!(state.inner.datagrams().poll_recv(waiter)); + Poll::Ready(datagram.map_err(|err| state.lost_as(err))) + }) } /// Wait for the connection to be closed for any reason @@ -387,24 +358,12 @@ impl Connection { /// application layer. Cases that might be routine include [`Error::LocallyClosed`] /// and [`Error::ApplicationClosed`]. pub async fn closed(&self) -> Error { - { - let conn = self.0.state.lock(); - if let Some(error) = conn.error.as_ref() { - return error.clone(); - } - // Construct the future while the lock is held to ensure we can't miss a wakeup if - // the `Notify` is signaled immediately after we release the lock. `await` it after - // the lock guard is out of scope. - self.0.shared.closed.notified() - } - .await; - self.0 - .state - .lock() - .error - .as_ref() - .expect("closed without an error") - .clone() + kio::wait(|waiter| { + let mut state = self.0.state.lock(); + let err = ready!(state.inner.poll_closed(waiter)); + Poll::Ready(state.lost_as(err)) + }) + .await } /// Whether the connection is closed, and why. @@ -416,7 +375,7 @@ impl Connection { /// /// Note: when the connection is closed, `connection.close_reason().is_some()` will always be true. pub fn close_reason(&self) -> Option { - self.0.state.lock().error.clone() + self.0.state.lock().error() } /// Close the connection immediately. @@ -450,7 +409,7 @@ impl Connection { /// [`close()`]: Connection::close pub fn close(&self, error_code: VarInt, reason: &str) { let conn = &mut *self.0.state.lock(); - conn.close(error_code, reason.to_string(), &self.0.shared); + conn.close(error_code, reason.to_string()); } /// Transmit `data` as an unreliable application datagram, scheduled by `options` @@ -469,16 +428,14 @@ impl Connection { pub fn send_datagram(&self, data: impl Buf, options: send::Options<'_>) -> Result<(), send::Error> { let options = options.to_quic(&self.0.0); let conn = &mut *self.0.state.lock(); - if let Some(x) = &conn.error { - return Err(send::Error::Connection(x.clone())); + if let Some(x) = conn.error() { + return Err(send::Error::Connection(x)); } match conn.inner.datagrams().send(data, options) { - Ok(()) => {} - Err(moq_quic::send::Error::TooLarge) => return Err(send::Error::TooLarge), - Err(_) => return Err(send::Error::Unsupported), + Ok(()) => Ok(()), + Err(moq_quic::send::Error::TooLarge) => Err(send::Error::TooLarge), + Err(_) => Err(send::Error::Unsupported), } - conn.wake(); - Ok(()) } /// Compute the maximum size of datagrams that may be passed to [`send_datagram()`]. @@ -535,10 +492,11 @@ impl Connection { self.0.state.lock().inner.streams().send_streams() } - /// How many streams have `send::Stream::stopped` waiters registered, to check none leak + /// Whether send stream `id` was released, so moq-quic no longer keeps how it ended #[cfg(test)] - pub(crate) fn stopped_waiters(&self) -> usize { - self.0.state.lock().stopped.len() + pub(crate) fn send_released(&self, id: StreamId) -> bool { + let mut state = self.0.state.lock(); + matches!(state.inner.send_stream(id).end(), Err(moq_quic::send::Error::Closed)) } /// The wire counters the driver updates as it sends and receives. @@ -585,172 +543,38 @@ impl Connection { /// See [`moq_quic::connection::Connection::set_limits`] pub fn set_limits(&self, limits: moq_quic::Limits) -> Result<(), moq_quic::LimitsError> { - let mut conn = self.0.state.lock(); - conn.inner.set_limits(limits)?; - // May need to send MAX_DATA or MAX_STREAMS to make progress - conn.wake(); - Ok(()) - } -} - -pin_project! { - /// Future produced by [`Connection::open_uni`] - struct OpenUni<'a> { - conn: &'a ConnectionRef, - options: send::Options<'a>, - #[pin] - notify: Notified<'a>, + self.0.state.lock().inner.set_limits(limits) } } -impl Future for OpenUni<'_> { - type Output = Result; - fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll { - let this = self.project(); - let (conn, id) = ready!(poll_open(ctx, this.conn, *this.options, this.notify, Dir::Uni))?; - Poll::Ready(Ok(send::Stream::new(conn, id))) - } -} - -pin_project! { - /// Future produced by [`Connection::open_bi`] - struct OpenBi<'a> { - conn: &'a ConnectionRef, - options: send::Options<'a>, - #[pin] - notify: Notified<'a>, - } -} - -impl Future for OpenBi<'_> { - type Output = Result<(send::Stream, recv::Stream), Error>; - fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll { - let this = self.project(); - let (conn, id) = ready!(poll_open(ctx, this.conn, *this.options, this.notify, Dir::Bi))?; - - Poll::Ready(Ok((send::Stream::new(conn.clone(), id), recv::Stream::new(conn, id)))) - } -} - -fn poll_open<'a>( - ctx: &mut Context<'_>, - conn: &'a ConnectionRef, +/// Open a stream in `dir`, or wait for the peer to allow another +fn poll_open( + waiter: &kio::Waiter, + conn: &ConnectionRef, options: send::Options<'_>, - mut notify: Pin<&mut Notified<'a>>, dir: Dir, ) -> Poll> { let options = options.to_quic(&conn.0); let mut state = conn.state.lock(); - if let Some(e) = &state.error { - return Poll::Ready(Err(e.clone())); - } else if let Some(id) = state.inner.streams().open(dir, options) { - drop(state); // Release the lock so clone can take it - return Poll::Ready(Ok((conn.clone(), id))); - } - loop { - match notify.as_mut().poll(ctx) { - // `state` lock ensures we didn't race with readiness - Poll::Pending => return Poll::Pending, - // Spurious wakeup, get a new future - Poll::Ready(()) => notify.set(conn.shared.stream_budget_available[dir as usize].notified()), - } - } -} - -pin_project! { - /// Future produced by [`Connection::accept_uni`] - struct AcceptUni<'a> { - conn: &'a ConnectionRef, - #[pin] - notify: Notified<'a>, - } -} - -impl Future for AcceptUni<'_> { - type Output = Result; - - fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll { - let this = self.project(); - let (conn, id) = ready!(poll_accept(ctx, this.conn, this.notify, Dir::Uni))?; - Poll::Ready(Ok(recv::Stream::new(conn, id))) - } -} - -pin_project! { - /// Future produced by [`Connection::accept_bi`] - struct AcceptBi<'a> { - conn: &'a ConnectionRef, - #[pin] - notify: Notified<'a>, - } -} - -impl Future for AcceptBi<'_> { - type Output = Result<(send::Stream, recv::Stream), Error>; - - fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll { - let this = self.project(); - let (conn, id) = ready!(poll_accept(ctx, this.conn, this.notify, Dir::Bi))?; - Poll::Ready(Ok((send::Stream::new(conn.clone(), id), recv::Stream::new(conn, id)))) - } + let id = match ready!(state.inner.streams().poll_open(waiter, dir, options)) { + Ok(id) => id, + Err(err) => return Poll::Ready(Err(state.lost_as(err))), + }; + drop(state); // Release the lock so clone can take it + Poll::Ready(Ok((conn.clone(), id))) } -fn poll_accept<'a>( - ctx: &mut Context<'_>, - conn: &'a ConnectionRef, - mut notify: Pin<&mut Notified<'a>>, - dir: Dir, -) -> Poll> { +/// Accept the peer's next stream in `dir`, or wait for one +fn poll_accept(waiter: &kio::Waiter, conn: &ConnectionRef, dir: Dir) -> Poll> { let mut state = conn.state.lock(); - // Check for incoming streams before checking `state.error` so that already-received streams, - // which are necessarily finite, can be drained from a closed connection. - if let Some(id) = state.inner.streams().accept(dir) { - state.wake(); // To send additional stream ID credit - drop(state); // Release the lock so clone can take it - return Poll::Ready(Ok((conn.clone(), id))); - } else if let Some(e) = &state.error { - return Poll::Ready(Err(e.clone())); - } - loop { - match notify.as_mut().poll(ctx) { - // `state` lock ensures we didn't race with readiness - Poll::Pending => return Poll::Pending, - // Spurious wakeup, get a new future - Poll::Ready(()) => notify.set(conn.shared.stream_incoming[dir as usize].notified()), - } - } -} - -pin_project! { - /// Future produced by [`Connection::read_datagram`] - struct ReadDatagram<'a> { - conn: &'a ConnectionRef, - #[pin] - notify: Notified<'a>, - } -} - -impl Future for ReadDatagram<'_> { - type Output = Result; - fn poll(self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll { - let mut this = self.project(); - let mut state = this.conn.state.lock(); - // Check for buffered datagrams before checking `state.error` so that already-received - // datagrams, which are necessarily finite, can be drained from a closed connection. - if let Some(x) = state.inner.datagrams().recv() { - return Poll::Ready(Ok(x)); - } else if let Some(e) = &state.error { - return Poll::Ready(Err(e.clone())); - } - loop { - match this.notify.as_mut().poll(ctx) { - // `state` lock ensures we didn't race with readiness - Poll::Pending => return Poll::Pending, - // Spurious wakeup, get a new future - Poll::Ready(()) => this.notify.set(this.conn.shared.datagram_received.notified()), - } - } - } + // moq-quic yields the streams received before checking for the connection error, so they, + // which are necessarily finite, drain from a closed connection. + let id = match ready!(state.inner.streams().poll_accept(waiter, dir)) { + Ok(id) => id, + Err(err) => return Poll::Ready(Err(state.lost_as(err))), + }; + drop(state); // Release the lock so clone can take it + Poll::Ready(Ok((conn.clone(), id))) } #[derive(Debug)] @@ -784,7 +608,7 @@ impl Drop for ConnectionRef { // not, we can't do any harm. If there were any streams being opened, then either // the connection will be closed for an unrelated reason or a fresh reference will // be constructed for the newly opened stream. - conn.implicit_close(&self.shared); + conn.implicit_close(); } } } @@ -804,14 +628,6 @@ pub(crate) struct ConnectionInner { #[derive(Debug, Default)] pub(crate) struct Shared { - /// Notified when new streams may be locally initiated due to an increase in stream ID flow - /// control budget - stream_budget_available: [Notify; 2], - /// Notified when the peer has initiated a new stream - stream_incoming: [Notify; 2], - datagram_received: Notify, - closed: Notify, - connected: Arc, /// Number of live handles that can used to initiate or handle I/O; excludes the driver ref_count: AtomicUsize, /// Refreshed by every driver poll, so readers never wake the driver to sample. @@ -826,19 +642,15 @@ pub(crate) struct Shared { pub(crate) struct State { pub(crate) inner: moq_quic::connection::Connection, - driver: Option, handle: ConnectionHandle, - on_handshake_data: Option>, - connected: bool, timer: Option, timer_deadline: Option, conn_events: mpsc::UnboundedReceiver, endpoint_events: mpsc::UnboundedSender<(ConnectionHandle, EndpointEvent)>, - pub(crate) blocked_writers: FxHashMap, - pub(crate) blocked_readers: FxHashMap, - pub(crate) stopped: FxHashMap>, - /// Always set to Some before the connection becomes drained - pub(crate) error: Option, + /// Why the connection failed, when moq-quic did not see it fail: the endpoint driver went away + /// + /// moq-quic takes no foreign error, so this wins over its reason when handles report the loss. + external: Option, /// The endpoint's socket, sent through without keeping it open slot: Arc, send_buffer: Vec, @@ -852,30 +664,25 @@ impl State { handle: ConnectionHandle, endpoint_events: mpsc::UnboundedSender<(ConnectionHandle, EndpointEvent)>, conn_events: mpsc::UnboundedReceiver, - on_handshake_data: oneshot::Sender<()>, slot: Arc, ) -> Self { Self { inner, - driver: None, handle, - on_handshake_data: Some(on_handshake_data), - connected: false, timer: None, timer_deadline: None, conn_events, endpoint_events, - blocked_writers: FxHashMap::default(), - blocked_readers: FxHashMap::default(), - stopped: FxHashMap::default(), - error: None, + external: None, slot, send_buffer: Vec::new(), buffered_transmit: None, } } - fn drive_transmit(&mut self, cx: &mut Context<'_>) -> io::Result { + /// Send what moq-quic has, parking `waiter` until the application queues more once it has + /// nothing, or `cx` on the socket once it is full + fn drive_transmit(&mut self, waiter: &kio::Waiter, cx: &mut Context<'_>) -> io::Result { let now = Instant::now(); let mut transmits = 0; @@ -888,15 +695,18 @@ impl State { None => { self.send_buffer.clear(); self.send_buffer.reserve(self.inner.current_mtu() as usize); - match self.inner.poll_transmit(now, max_datagrams, &mut self.send_buffer) { - Some(t) => { + match self + .inner + .poll_transmit(waiter, now, max_datagrams, &mut self.send_buffer) + { + Poll::Ready(t) => { transmits += match t.segment_size { None => 1, Some(s) => t.size.div_ceil(s), // round up }; t } - None => break, + Poll::Pending => break, } } }; @@ -935,7 +745,7 @@ impl State { } /// If this returns `Err`, the endpoint is dead, so the driver should exit immediately. - fn process_conn_events(&mut self, shared: &Shared, cx: &mut Context<'_>) -> Result<(), Error> { + fn process_conn_events(&mut self, cx: &mut Context<'_>) -> Result<(), Error> { loop { match self.conn_events.poll_recv(cx) { Poll::Ready(Some(ConnectionEvent::Rebind)) => { @@ -945,7 +755,7 @@ impl State { self.inner.handle_event(event); } Poll::Ready(Some(ConnectionEvent::Close { reason, error_code })) => { - self.close(error_code, reason, shared); + self.close(error_code, reason); } Poll::Ready(None) => { return Err(Error::TransportError(TransportError::new( @@ -960,47 +770,9 @@ impl State { } } - fn forward_app_events(&mut self, shared: &Shared) { - while let Some(event) = self.inner.poll() { - use moq_quic::Event::*; - match event { - HandshakeDataReady => { - if let Some(x) = self.on_handshake_data.take() { - let _ = x.send(()); - } - } - Connected => { - self.connected = true; - shared.connected.notify_waiters(); - } - - ConnectionLost { reason } => { - self.terminate(reason, shared); - } - Stream(StreamEvent::Writable { id }) => wake_stream(id, &mut self.blocked_writers), - Stream(StreamEvent::Opened { dir: Dir::Uni }) => { - shared.stream_incoming[Dir::Uni as usize].notify_waiters(); - } - Stream(StreamEvent::Opened { dir: Dir::Bi }) => { - shared.stream_incoming[Dir::Bi as usize].notify_waiters(); - } - DatagramReceived => { - shared.datagram_received.notify_waiters(); - } - Stream(StreamEvent::Readable { id }) => wake_stream(id, &mut self.blocked_readers), - Stream(StreamEvent::Available { dir }) => { - // Might mean any number of streams are ready, so we wake up everyone - shared.stream_budget_available[dir as usize].notify_waiters(); - } - Stream(StreamEvent::Finished { id }) => self.resolve_stopped(id, None), - Stream(StreamEvent::Stopped { id, error_code }) => { - self.resolve_stopped(id, Some(error_code)); - wake_stream(id, &mut self.blocked_writers); - } - - _ => {} - } - } + /// Drain moq-quic's events, which no handle waits on: moq-quic wakes its waiters itself. + fn drain_events(&mut self) { + while self.inner.poll().is_some() {} } fn drive_timer(&mut self, cx: &mut Context<'_>) -> bool { @@ -1042,49 +814,45 @@ impl State { true } - /// Wake up a blocked `Driver` task to process I/O - pub(crate) fn wake(&mut self) { - if let Some(x) = self.driver.take() { - x.wake(); - } + /// Why the connection was lost, if it was: the driver's external reason, or moq-quic's + pub(crate) fn error(&self) -> Option { + self.external.clone().or_else(|| self.inner.error().cloned()) } - /// Used to wake up all blocked futures when the connection becomes closed for any reason - fn terminate(&mut self, reason: Error, shared: &Shared) { - self.error = Some(reason.clone()); - if let Some(x) = self.on_handshake_data.take() { - let _ = x.send(()); - } - wake_all(&mut self.blocked_writers); - wake_all(&mut self.blocked_readers); - shared.stream_budget_available[Dir::Uni as usize].notify_waiters(); - shared.stream_budget_available[Dir::Bi as usize].notify_waiters(); - shared.stream_incoming[Dir::Uni as usize].notify_waiters(); - shared.stream_incoming[Dir::Bi as usize].notify_waiters(); - shared.datagram_received.notify_waiters(); - wake_all_notify(&mut self.stopped); - shared.closed.notify_waiters(); - shared.connected.notify_waiters(); + /// Whether the connection was lost, without cloning why + pub(crate) fn is_lost(&self) -> bool { + self.external.is_some() || self.inner.error().is_some() } - fn close(&mut self, error_code: VarInt, reason: String, shared: &Shared) { - self.inner.close(Instant::now(), error_code, reason); - // Fails pending operations now, with the close's own code and reason. - self.forward_app_events(shared); - self.wake(); + /// The error handles report for a loss moq-quic reported as `err` + fn lost_as(&self, err: Error) -> Error { + self.external.clone().unwrap_or(err) } - /// Close for a reason other than the application's explicit request - pub(crate) fn implicit_close(&mut self, shared: &Shared) { - self.close(0u32.into(), String::new(), shared); + /// The error handles report for a loss moq-quic reported without a reason, as a stream's + /// `ConnectionLost` + pub(crate) fn lost(&self) -> Error { + self.error().expect("moq-quic reported a loss without a reason") } - /// Resolves the stream's `stopped` waiters with the peer's STOP_SENDING code, or `None` - pub(crate) fn resolve_stopped(&mut self, id: StreamId, outcome: Option) { - if let Some(waiters) = self.stopped.remove(&id) { - let _ = waiters.outcome.set(outcome); - waiters.notify.notify_waiters(); - } + /// Fail the connection for `reason`, which moq-quic did not see: the endpoint driver went away + /// + /// The reason is kept to win over moq-quic's, and a local close makes moq-quic wake every + /// waiter. The driver exits right after, so the close is never sent. + fn terminate(&mut self, reason: Error) { + let message = reason.to_string(); + self.external = Some(reason); + self.inner.close(Instant::now(), 0u32.into(), message); + } + + fn close(&mut self, error_code: VarInt, reason: String) { + // moq-quic fails pending operations now, with the close's own code and reason. + self.inner.close(Instant::now(), error_code, reason); + } + + /// Close for a reason other than the application's explicit request + pub(crate) fn implicit_close(&mut self) { + self.close(0u32.into(), String::new()); } } @@ -1103,29 +871,6 @@ impl fmt::Debug for State { } } -fn wake_stream(stream_id: StreamId, wakers: &mut FxHashMap) { - if let Some(waker) = wakers.remove(&stream_id) { - waker.wake(); - } -} - -fn wake_all(wakers: &mut FxHashMap) { - wakers.drain().for_each(|(_, waker)| waker.wake()) -} - -/// The `send::Stream::stopped` futures waiting on one stream -#[derive(Debug, Default)] -pub(crate) struct StopWaiters { - pub(crate) notify: Notify, - /// What a STOP_SENDING, a delivered finish, or a local reset settled, kept here because the - /// stream may be freed before a woken waiter polls. Once set, nothing later counts. - pub(crate) outcome: OnceLock>, -} - -fn wake_all_notify(wakers: &mut FxHashMap>) { - wakers.drain().for_each(|(_, waiters)| waiters.notify.notify_waiters()) -} - /// The maximum amount of datagrams which will be produced in a single `drive_transmit` call /// /// This limits the amount of CPU resources consumed by datagram generation, diff --git a/rs/moq-tokio/src/quic/driver/recv.rs b/rs/moq-tokio/src/quic/driver/recv.rs index 300aeebbe06..4673f98b118 100644 --- a/rs/moq-tokio/src/quic/driver/recv.rs +++ b/rs/moq-tokio/src/quic/driver/recv.rs @@ -1,14 +1,16 @@ //! Incoming streams: their handles, leases, and errors. use std::{ - future::poll_fn, io, pin::Pin, task::{Context, Poll, ready}, }; use bytes::{Buf, BufMut, Bytes}; -use moq_quic::{connection::Error as ConnectionError, recv::Chunks}; +use moq_quic::{ + connection::Error as ConnectionError, + recv::{Chunk, Chunks}, +}; use thiserror::Error; use tokio::io::ReadBuf; @@ -46,6 +48,15 @@ use crate::quic::driver::{VarInt, connection::ConnectionRef, stream}; /// [`Connection::accept_bi`]: crate::quic::driver::connection::Connection::accept_bi #[derive(Debug)] pub struct Stream { + reader: Reader, + /// Holds the waiter of a read polled from a [`Context`], by + /// [`AsyncRead`](tokio::io::AsyncRead) or [`Self::poll_lease`], across polls + park: kio::Park, +} + +/// The state a [`Stream`]'s reads borrow, apart from its park +#[derive(Debug)] +struct Reader { conn: ConnectionRef, id: stream::Id, all_data_read: bool, @@ -57,11 +68,14 @@ pub struct Stream { impl Stream { pub(crate) fn new(conn: ConnectionRef, id: stream::Id) -> Self { Self { - conn, - id, - all_data_read: false, - reset: None, - drop_stop: VarInt::from_u32(0), + reader: Reader { + conn, + id, + all_data_read: false, + reset: None, + drop_stop: VarInt::from_u32(0), + }, + park: kio::Park::default(), } } @@ -72,7 +86,7 @@ impl Stream { /// /// This operation is cancel-safe. pub async fn read(&mut self, buf: &mut impl BufMut) -> Result, Error> { - poll_fn(|cx| self.poll_read_buf(cx, buf)).await + kio::wait(|waiter| self.reader.poll_read_buf(waiter, buf)).await } /// Fill `buf` with data from the stream @@ -97,11 +111,12 @@ impl Stream { /// /// This operation is cancel-safe. pub async fn read_chunk(&mut self, max: usize) -> Result, Error> { - poll_fn(|cx| { - self.poll_read_generic(cx, |chunks| match chunks.next(max) { - Ok(Some(chunk)) => ReadStatus::Readable(chunk.bytes), - res => (None, res.err()).into(), - }) + kio::wait(|waiter| { + self.reader + .poll_read_generic(waiter, |chunks, waiter| match poll_next(chunks, waiter, max) { + Ok(Some(chunk)) => ReadStatus::Readable(chunk.bytes), + res => (None, res.err()).into(), + }) }) .await } @@ -126,9 +141,9 @@ impl Stream { /// Lease the stream's unreleased bytes once at least `want` have arrived in order, or the /// stream ended. See [`Self::poll_lease()`]. pub async fn lease(&mut self, want: usize) -> Result, Error> { - let inner = poll_fn(|cx| self.poll_take(cx, want)).await?; + let inner = kio::wait(|waiter| self.reader.poll_take(waiter, want)).await?; Ok(Lease { - stream: self, + reader: &mut self.reader, inner: Some(inner), }) } @@ -150,72 +165,37 @@ impl Stream { /// /// Do not mix with the plain reads on one stream, which skip the leased bytes. pub fn poll_lease(&mut self, cx: &mut Context<'_>, want: usize) -> Poll, Error>> { - let inner = ready!(self.poll_take(cx, want))?; + let waiter = self.park.hold(cx); + let inner = ready!(self.reader.poll_take(waiter, want))?; Poll::Ready(Ok(Lease { - stream: self, + reader: &mut self.reader, inner: Some(inner), })) } - /// Take the connection's lease, which [`Lease`] wraps once this returns: the borrow of `self` - /// it holds cannot start inside a poll that may return `Pending`. - fn poll_take(&mut self, cx: &mut Context<'_>, want: usize) -> Poll> { - // moq-quic freed the stream when it reported the reset. - if let Some(code) = self.reset { - return Poll::Ready(Err(Error::Reset(code))); - } - let mut conn = self.conn.state.lock(); - let (result, transmit) = conn.inner.recv_stream(self.id).lease(want); - if transmit.should_transmit() { - conn.wake(); - } - let err = match result { - Ok(lease) => return Poll::Ready(Ok(lease)), - Err(moq_quic::recv::Error::Blocked) => { - if let Some(x) = &conn.error { - return Poll::Ready(Err(Error::Connection(x.clone()))); - } - conn.blocked_readers.insert(self.id, cx.waker().clone()); - return Poll::Pending; - } - Err(err) => err, - }; - conn.blocked_readers.remove(&self.id); - let closed = conn.error.clone(); - drop(conn); - Poll::Ready(Err(lease_error(err, closed))) - } - /// Rank this stream for eviction: when the connection window is stuck, the stream with the /// lowest priority goes first, and the newest of those that tie. pub fn set_priority(&mut self, priority: i64) { - let mut conn = self.conn.state.lock(); - conn.inner.recv_stream(self.id).set_priority(priority); - } - - fn poll_read_buf(&mut self, cx: &mut Context<'_>, buf: &mut impl BufMut) -> Poll, Error>> { - if !buf.has_remaining_mut() { - return Poll::Ready(Ok(Some(0))); - } - self.poll_read_generic(cx, |chunks| fill(chunks, buf)) + let mut conn = self.reader.conn.state.lock(); + conn.inner.recv_stream(self.reader.id).set_priority(priority); } /// Stop accepting data /// /// Discards unread data and notifies the peer to stop transmitting. pub fn stop(mut self, error_code: VarInt) { - self.drop_stop = error_code; + self.reader.drop_stop = error_code; } /// The identity of this stream pub fn id(&self) -> stream::Id { - self.id + self.reader.id } /// Stop with `code`, instead of code 0, if dropped before the end #[cfg(feature = "quic")] pub(crate) fn set_drop_stop(&mut self, code: VarInt) { - self.drop_stop = code; + self.reader.drop_stop = code; } /// Completes when the stream has been reset by the peer or otherwise closed @@ -228,41 +208,62 @@ impl Stream { /// /// This operation is cancel-safe. pub async fn received_reset(&mut self) -> Result, Error> { - poll_fn(|cx| self.poll_received_reset(cx)).await + kio::wait(|waiter| self.reader.poll_received_reset(waiter)).await } /// Poll form of [`Self::received_reset`]. It takes no bytes, so a lease in progress is /// left as it was. pub(crate) fn poll_received_reset(&mut self, cx: &mut Context<'_>) -> Poll, Error>> { + let waiter = self.park.hold(cx); + self.reader.poll_received_reset(waiter) + } +} + +impl Reader { + /// Take the connection's lease, which [`Lease`] wraps once this returns: the borrow of `self` + /// it holds cannot start inside a poll that may return `Pending`. + fn poll_take(&mut self, waiter: &kio::Waiter, want: usize) -> Poll> { + // moq-quic freed the stream when it reported the reset. + if let Some(code) = self.reset { + return Poll::Ready(Err(Error::Reset(code))); + } let mut conn = self.conn.state.lock(); + // moq-quic wakes the driver for any credit the lease returned. + let (result, _) = conn.inner.recv_stream(self.id).poll_lease(waiter, want); + let err = match ready!(result) { + Ok(lease) => return Poll::Ready(Ok(lease)), + Err(err) => err, + }; + let closed = conn.error(); + drop(conn); + Poll::Ready(Err(lease_error(err, closed))) + } + + fn poll_read_buf(&mut self, waiter: &kio::Waiter, buf: &mut impl BufMut) -> Poll, Error>> { + if !buf.has_remaining_mut() { + return Poll::Ready(Ok(Some(0))); + } + self.poll_read_generic(waiter, |chunks, waiter| fill(chunks, waiter, buf)) + } + fn poll_received_reset(&mut self, waiter: &kio::Waiter) -> Poll, Error>> { if let Some(code) = self.reset { return Poll::Ready(Ok(Some(code))); } - - match conn.inner.recv_stream(self.id).received_reset() { - // Evicted, so its bytes are gone: nothing frees the window until it is stopped. - Err(moq_quic::recv::Error::Evicted) => Poll::Ready(Err(Error::Evicted)), - Err(_) => Poll::Ready(Ok(None)), - Ok(Some(error_code)) => { - // Stream state has just now been freed, so the connection may need to issue new - // stream ID flow control credit, and later reads fail with the cached code. - self.reset = Some(error_code); - conn.wake(); - Poll::Ready(Ok(Some(error_code))) - } - Ok(None) => { - if let Some(e) = &conn.error { - return Poll::Ready(Err(e.clone().into())); + let mut conn = self.conn.state.lock(); + Poll::Ready( + match ready!(conn.inner.recv_stream(self.id).poll_received_reset(waiter)) { + Ok(error_code) => { + // Stream state has just now been freed, so later reads fail with the cached code. + self.reset = Some(error_code); + Ok(Some(error_code)) } - // Resets always notify readers, since a reset is an immediate read error. We - // could introduce a dedicated channel to reduce the risk of spurious wakeups, - // but that increased complexity is probably not justified, as an application - // that is expecting a reset is not likely to receive large amounts of data. - conn.blocked_readers.insert(self.id, cx.waker().clone()); - Poll::Pending - } - } + // Evicted, so its bytes are gone: nothing frees the window until it is stopped. + Err(moq_quic::recv::Error::Evicted) => Err(Error::Evicted), + Err(moq_quic::recv::Error::ConnectionLost) => Err(Error::Connection(conn.lost())), + Err(_) => Ok(None), + }, + ) } /// Handle common logic related to reading out of a receive stream @@ -270,10 +271,12 @@ impl Stream { /// This takes an `FnMut` closure that takes care of the actual reading process, matching /// the detailed read semantics for the calling function with a particular return type. /// The closure can read from the passed `&mut Chunks` and has to return the status after - /// reading: the amount of data read, and the status after the final read call. - fn poll_read_generic(&mut self, cx: &mut Context<'_>, mut read_fn: T) -> Poll, Error>> + /// reading: the amount of data read, and the status after the final read call. It reports + /// [`Blocked`](moq_quic::recv::Error::Blocked) with nothing read only once it registered + /// the waiter, as [`poll_next`] does. + fn poll_read_generic(&mut self, waiter: &kio::Waiter, mut read_fn: T) -> Poll, Error>> where - T: FnMut(&mut Chunks<'_>) -> ReadStatus, + T: FnMut(&mut Chunks<'_>, &kio::Waiter) -> ReadStatus, { use moq_quic::recv::Error::{Blocked, Reset}; @@ -289,17 +292,15 @@ impl Stream { Some(code) => ReadStatus::Failed(None, Reset(code)), None => { let mut recv = conn.inner.recv_stream(self.id); - let (status, transmit) = match recv.read() { + match recv.read() { Ok(mut chunks) => { - let status = read_fn(&mut chunks); - (status, chunks.finalize().should_transmit()) + let status = read_fn(&mut chunks, waiter); + // moq-quic wakes the driver for the credit the read returned. + let _ = chunks.finalize(); + status } - Err(err) => (ReadStatus::Failed(None, err), false), - }; - if transmit { - conn.wake(); + Err(err) => ReadStatus::Failed(None, err), } - status } }; @@ -307,30 +308,21 @@ impl Stream { ReadStatus::Readable(read) => Poll::Ready(Ok(Some(read))), ReadStatus::Finished(read) => { self.all_data_read = true; - // Stream is dead, cleanup stale waker - conn.blocked_readers.remove(&self.id); Poll::Ready(Ok(read)) } ReadStatus::Failed(Some(read), Blocked) => Poll::Ready(Ok(Some(read))), - ReadStatus::Failed(None, Blocked) => { - if let Some(x) = &conn.error { - return Poll::Ready(Err(Error::Connection(x.clone()))); - } - conn.blocked_readers.insert(self.id, cx.waker().clone()); - Poll::Pending - } + // moq-quic registered the waiter. + ReadStatus::Failed(None, Blocked) => Poll::Pending, ReadStatus::Failed(read, Reset(error_code)) => { self.reset = Some(error_code); if read.is_some() { return Poll::Ready(Ok(read)); } self.all_data_read = true; - // Stream is dead, cleanup stale waker - conn.blocked_readers.remove(&self.id); Poll::Ready(Err(Error::Reset(error_code))) } ReadStatus::Failed(_, err) => { - let error = conn.error.clone(); + let error = conn.error(); drop(conn); Poll::Ready(Err(read_error(err, error))) } @@ -338,8 +330,22 @@ impl Stream { } } +/// [`Chunks::poll_next`], reporting a registered wait as [`Blocked`](moq_quic::recv::Error::Blocked) +fn poll_next( + chunks: &mut Chunks<'_>, + waiter: &kio::Waiter, + max: usize, +) -> Result, moq_quic::recv::Error> { + match chunks.poll_next(waiter, max) { + Poll::Ready(result) => result, + Poll::Pending => Err(moq_quic::recv::Error::Blocked), + } +} + /// Copy from `chunks` until `buf` is full or nothing more is readable, returning how much. -fn fill(chunks: &mut Chunks<'_>, buf: &mut impl BufMut) -> ReadStatus { +/// +/// It waits on `waiter` only when it read nothing. +fn fill(chunks: &mut Chunks<'_>, waiter: &kio::Waiter, buf: &mut impl BufMut) -> ReadStatus { let mut read = 0; loop { if !buf.has_remaining_mut() { @@ -347,7 +353,12 @@ fn fill(chunks: &mut Chunks<'_>, buf: &mut impl BufMut) -> ReadStatus { return ReadStatus::Readable(read); } - match chunks.next(buf.remaining_mut()) { + let max = buf.remaining_mut(); + let next = match read { + 0 => poll_next(chunks, waiter, max), + _ => chunks.next(max), + }; + match next { Ok(Some(chunk)) => { buf.put_slice(&chunk.bytes); read += chunk.bytes.len(); @@ -385,7 +396,7 @@ fn read_error(err: moq_quic::recv::Error, closed: Option) -> Er /// time. #[derive(Debug)] pub struct Lease<'a> { - stream: &'a mut Stream, + reader: &'a mut Reader, /// Taken only to hand it back. inner: Option, } @@ -412,10 +423,8 @@ impl Lease<'_> { /// Return the credit of every byte read so far, by this lease or an earlier one. pub fn release(mut self) { let lease = self.inner.take().expect("only taken to hand it back"); - let mut conn = self.stream.conn.state.lock(); - if conn.inner.recv_stream(self.stream.id).release(lease).should_transmit() { - conn.wake(); - } + let mut conn = self.reader.conn.state.lock(); + let _ = conn.inner.recv_stream(self.reader.id).release(lease); } } @@ -441,10 +450,9 @@ impl Buf for Lease<'_> { impl Drop for Lease<'_> { fn drop(&mut self) { if let Some(lease) = self.inner.take() - && let Some(mut conn) = self.stream.conn.state.lock_for_drop() - && conn.inner.recv_stream(self.stream.id).restore(lease).should_transmit() + && let Some(mut conn) = self.reader.conn.state.lock_for_drop() { - conn.wake(); + let _ = conn.inner.recv_stream(self.reader.id).restore(lease); } } } @@ -466,8 +474,10 @@ impl From<(Option, Option)> for ReadStatus { impl tokio::io::AsyncRead for Stream { fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll> { + let this = self.get_mut(); + let waiter = this.park.hold(cx); let mut unfilled = buf.initialize_unfilled(); - let read = ready!(self.get_mut().poll_read_buf(cx, &mut unfilled))?; + let read = ready!(this.reader.poll_read_buf(waiter, &mut unfilled))?; buf.advance(read.unwrap_or(0)); Poll::Ready(Ok(())) } @@ -475,20 +485,16 @@ impl tokio::io::AsyncRead for Stream { impl Drop for Stream { fn drop(&mut self) { - let Some(mut conn) = self.conn.state.lock_for_drop() else { + let reader = &self.reader; + let Some(mut conn) = reader.conn.state.lock_for_drop() else { return; }; - - // clean up any previously registered wakers - conn.blocked_readers.remove(&self.id); - - if self.all_data_read || conn.error.is_some() { + if reader.all_data_read || conn.is_lost() { return; } // Fails harmlessly once the stream is gone. - let _ = conn.inner.recv_stream(self.id).stop(self.drop_stop); - conn.wake(); + let _ = conn.inner.recv_stream(reader.id).stop(reader.drop_stop); } } diff --git a/rs/moq-tokio/src/quic/driver/send.rs b/rs/moq-tokio/src/quic/driver/send.rs index bc2b23f835d..70388d082af 100644 --- a/rs/moq-tokio/src/quic/driver/send.rs +++ b/rs/moq-tokio/src/quic/driver/send.rs @@ -1,11 +1,11 @@ //! Outgoing streams and datagrams: their handles, scheduling, and errors. use std::{ - future::{Future, poll_fn}, + future::Future, io, pin::Pin, - sync::Arc, - task::{Context, Poll}, + sync::{Arc, OnceLock}, + task::{Context, Poll, ready}, }; use bytes::Buf; @@ -14,7 +14,7 @@ use thiserror::Error; use crate::quic::driver::{ VarInt, - connection::{ConnectionInner, ConnectionRef, State}, + connection::{ConnectionInner, ConnectionRef}, stream, }; @@ -32,7 +32,11 @@ use crate::quic::driver::{ /// perform multiple writes internally before succeeding. Each `write` method documents whether it is /// cancel-safe. #[derive(Debug)] -pub struct Stream(Half); +pub struct Stream { + half: Half, + /// Holds the waiter of an [`AsyncWrite`](tokio::io::AsyncWrite) write across polls + park: kio::Park, +} /// A stream [`Stream::finish`]ed, which keeps (re)transmitting its data until the peer /// acknowledges all of it or the connection closes @@ -45,8 +49,7 @@ pub struct Finished(Half); /// What [`Stream`] and [`Finished`] share. #[derive(Debug)] struct Half { - conn: ConnectionRef, - id: stream::Id, + stream: Arc, state: HalfState, } @@ -59,13 +62,32 @@ enum HalfState { Reset, } +/// The stream's ID, shared by its [`Half`] and the `stopped` futures, which may outlive it +/// +/// moq-quic forgets how a stream ended once its ID is released, so the last of them to drop +/// releases it. +#[derive(Debug)] +struct StreamRef { + conn: ConnectionRef, + id: stream::Id, + /// What a local reset settled for `stopped`: the peer's code if its stop arrived first, or + /// `None`. moq-quic still records a stop that arrives after the reset, which no longer counts. + reset: OnceLock>, +} + impl Stream { pub(crate) fn new(conn: ConnectionRef, id: stream::Id) -> Self { - Self(Half { - conn, - id, - state: HalfState::Open(None, VarInt::from_u32(0)), - }) + Self { + half: Half { + stream: Arc::new(StreamRef { + conn, + id, + reset: OnceLock::new(), + }), + state: HalfState::Open(None, VarInt::from_u32(0)), + }, + park: kio::Park::default(), + } } /// Write from `buf` into this stream, returning how many bytes were written @@ -79,7 +101,7 @@ impl Stream { /// /// This method is cancellation safe. If this does not resolve, no bytes were written. pub async fn write(&mut self, buf: &mut impl Buf) -> Result { - poll_fn(|cx| self.poll_write(cx, buf)).await + kio::wait(|waiter| self.half.poll_write(waiter, buf)).await } /// Write all of `buf` into this stream @@ -95,36 +117,13 @@ impl Stream { Ok(()) } - fn poll_write(&mut self, cx: &mut Context<'_>, buf: &mut impl Buf) -> Poll> { - assert!( - matches!(self.0.state, HalfState::Open(..)), - "wrote to a stream after shutting it down" - ); - let mut conn = self.0.conn.state.lock(); - if let Some(x) = &conn.error { - return Poll::Ready(Err(Error::Connection(x.clone()))); - } - match conn.inner.send_stream(self.0.id).write(buf) { - Ok(written) => { - conn.wake(); - Poll::Ready(Ok(written)) - } - Err(moq_quic::send::Error::Blocked) => { - conn.blocked_writers.insert(self.0.id, cx.waker().clone()); - Poll::Pending - } - Err(moq_quic::send::Error::Stopped(code)) => Poll::Ready(Err(Error::Stopped(code))), - Err(err) => unreachable!("an open stream failed to write: {err:?}"), - } - } - /// Notify the peer that no more data will ever be written to this stream /// /// [`Finished::stopped()`] waits for the peer to receive all of it, or learns that it /// stopped the stream first. pub fn finish(mut self) -> Finished { - self.0.finish(); - Finished(self.0) + self.half.finish(); + Finished(self.half) } /// Close the send stream immediately @@ -132,7 +131,7 @@ impl Stream { /// Locally buffered data is dropped, and previously transmitted data will no longer be /// retransmitted if lost. pub fn reset(mut self, error_code: VarInt) { - self.0.reset(None, error_code); + self.half.reset(None, error_code); } /// Close the send stream immediately, but still deliver the data before `offset` @@ -145,16 +144,17 @@ impl Stream { /// # Panics /// - when `offset` came from another stream pub fn reset_at(mut self, offset: Offset, error_code: VarInt) { - self.0.reset(Some(offset), error_code); + self.half.reset(Some(offset), error_code); } /// The offset just past the data written so far pub fn offset(&self) -> Offset { - let mut conn = self.0.conn.state.lock(); + let stream = &self.half.stream; + let mut conn = stream.conn.state.lock(); Offset { - inner: conn.inner.send_stream(self.0.id).offset(), - conn: self.0.conn.stable_id(), - id: self.0.id, + inner: conn.inner.send_stream(stream.id).offset(), + conn: stream.conn.stable_id(), + id: stream.id, } } @@ -162,9 +162,9 @@ impl Stream { #[cfg(feature = "quic")] pub(crate) fn set_drop_reset(&mut self, offset: Option, code: VarInt) { if let Some(offset) = &offset { - self.0.own(offset); + self.half.own(offset); } - self.0.state = HalfState::Open(offset, code); + self.half.state = HalfState::Open(offset, code); } /// Set the priority within the stream's send group, or session-wide when ungrouped @@ -175,12 +175,12 @@ impl Stream { /// transmitted. Using many different priority levels per connection may have a negative /// impact on performance. pub fn set_priority(&mut self, priority: i64) { - self.0.set_priority(priority); + self.half.set_priority(priority); } /// The priority of the send stream pub fn priority(&self) -> i64 { - self.0.priority() + self.half.priority() } /// Move the stream into a send group, or with `None` back to the ungrouped streams, which @@ -189,7 +189,7 @@ impl Stream { /// # Panics /// - when `group` belongs to another connection pub fn set_group(&mut self, group: Option<&Group>) { - self.0.set_group(group); + self.half.set_group(group); } /// Completes when the peer stops the stream, or the stream ends otherwise @@ -197,12 +197,12 @@ impl Stream { /// See [`Finished::stopped()`]. A stream reset or dropped before the peer stopped it /// resolves with `None`. pub fn stopped(&self) -> impl Future, Error>> + Send + Sync + 'static { - self.0.stopped() + self.half.stopped() } /// The identity of this stream pub fn id(&self) -> stream::Id { - self.0.id + self.half.stream.id } } @@ -255,7 +255,7 @@ impl Finished { /// The identity of this stream pub fn id(&self) -> stream::Id { - self.0.id + self.0.stream.id } } @@ -264,17 +264,34 @@ impl Half { /// there poisons the lock, so every handle's drop panics again and aborts the process. fn own(&self, offset: &Offset) { assert!( - offset.conn == self.conn.stable_id() && offset.id == self.id, + offset.conn == self.stream.conn.stable_id() && offset.id == self.stream.id, "offset used on another stream" ); } + fn poll_write(&mut self, waiter: &kio::Waiter, buf: &mut impl Buf) -> Poll> { + assert!( + matches!(self.state, HalfState::Open(..)), + "wrote to a stream after shutting it down" + ); + let mut conn = self.stream.conn.state.lock(); + if let Some(x) = conn.error() { + return Poll::Ready(Err(Error::Connection(x))); + } + Poll::Ready( + match ready!(conn.inner.send_stream(self.stream.id).poll_write(waiter, buf)) { + Ok(written) => Ok(written), + Err(moq_quic::send::Error::Stopped(code)) => Err(Error::Stopped(code)), + Err(moq_quic::send::Error::ConnectionLost) => Err(Error::Connection(conn.lost())), + Err(err) => unreachable!("an open stream failed to write: {err:?}"), + }, + ) + } + fn finish(&mut self) { - let mut conn = self.conn.state.lock(); + let mut conn = self.stream.conn.state.lock(); // A STOP_SENDING may have beaten us here, which `stopped` reports. - if conn.inner.send_stream(self.id).finish().is_ok() { - conn.wake(); - } + let _ = conn.inner.send_stream(self.stream.id).finish(); self.state = HalfState::Finished; } @@ -283,8 +300,8 @@ impl Half { self.own(offset); } self.state = HalfState::Reset; - let mut conn = self.conn.state.lock(); - let mut stream = conn.inner.send_stream(self.id); + let mut conn = self.stream.conn.state.lock(); + let mut stream = conn.inner.send_stream(self.stream.id); let stopped = match stream.end() { Ok(Some(End::Stopped(code))) => Some(code), _ => None, @@ -295,123 +312,98 @@ impl Half { }; // Fails harmlessly once the connection is gone. if reset.is_ok() { - conn.resolve_stopped(self.id, stopped); - conn.wake(); + let _ = self.stream.reset.set(stopped); } } fn set_priority(&mut self, priority: i64) { - let mut conn = self.conn.state.lock(); - let _ = conn.inner.send_stream(self.id).set_priority(priority); + let mut conn = self.stream.conn.state.lock(); + let _ = conn.inner.send_stream(self.stream.id).set_priority(priority); } fn priority(&self) -> i64 { - let mut conn = self.conn.state.lock(); - conn.inner.send_stream(self.id).priority().unwrap_or_default() + let mut conn = self.stream.conn.state.lock(); + conn.inner.send_stream(self.stream.id).priority().unwrap_or_default() } fn set_group(&mut self, group: Option<&Group>) { - let group = group.map(|group| group.id_on(&self.conn.0)); - let mut conn = self.conn.state.lock(); - let _ = conn.inner.send_stream(self.id).set_group(group); + let group = group.map(|group| group.id_on(&self.stream.conn.0)); + let mut conn = self.stream.conn.state.lock(); + let _ = conn.inner.send_stream(self.stream.id).set_group(group); } fn stopped(&self) -> impl Future, Error>> + Send + Sync + 'static { - let conn = self.conn.clone(); - let id = self.id; - // Join the waiters now rather than on the first poll, so a reset in between, which no - // stream event reports, still reaches this future. - let waiters = { - let mut conn = conn.state.lock(); - match ended(&mut conn, id) { - Some(output) => Err(output), - None => Ok(conn.stopped.entry(id).or_default().clone()), - } - }; - async move { - let mut waiters = match waiters { - Ok(waiters) => waiters, - Err(output) => return output, - }; - loop { - // The `Notify::notified` future needs to be created while the lock is being held, - // otherwise a wakeup could be missed if triggered inbetween releasing the lock - // and creating the future. - // The lock may only be held in a block without `await`s, otherwise the future - // becomes `!Send`. `Notify::notified` is lifetime-bound to `Notify`, therefore - // we need to declare `waiters` outside of the block, and update it inside. - { - let mut conn = conn.state.lock(); - if let Some(&stopped) = waiters.outcome.get() { - return Ok(stopped); - } - if let Some(output) = ended(&mut conn, id) { - return output; - } - - // A wakeup that resolved nothing removed the waiters; join the current ones - waiters = conn.stopped.entry(id).or_default().clone(); - waiters.notify.notified() - } - .await - } + // Holding the stream keeps its ID, and so how it ended, until this future drops. + let stream = self.stream.clone(); + kio::wait(move |waiter| stream.poll_stopped(waiter)) + } +} + +impl StreamRef { + /// Wait for the peer to stop the stream, or for it to end otherwise + fn poll_stopped(&self, waiter: &kio::Waiter) -> Poll, Error>> { + let mut conn = self.conn.state.lock(); + if let Some(&stopped) = self.reset.get() { + return Poll::Ready(Ok(stopped)); } + Poll::Ready(match ready!(conn.inner.send_stream(self.id).poll_end(waiter)) { + Ok(End::Stopped(code)) => Ok(Some(code)), + Ok(_) => Ok(None), + Err(moq_quic::send::Error::ConnectionLost) => Err(Error::Connection(conn.lost())), + // Only a released ID fails otherwise, which this stream holds until it drops. + Err(err) => unreachable!("a held send stream failed to end: {err:?}"), + }) } } -/// How a send stream ended, or `None` while it goes on -/// -/// A released stream's end is gone, but its handle settled the waiters when it let go. -fn ended(conn: &mut State, id: stream::Id) -> Option, Error>> { - match conn.inner.send_stream(id).end() { - Ok(Some(End::Stopped(code))) => Some(Ok(Some(code))), - Ok(Some(_)) => Some(Ok(None)), - Ok(None) | Err(_) => conn.error.clone().map(|error| Err(error.into())), +impl Drop for StreamRef { + fn drop(&mut self) { + if let Some(mut conn) = self.conn.state.lock_for_drop() { + conn.inner.send_stream(self.id).release(); + } } } impl Drop for Half { fn drop(&mut self) { - let Some(mut conn) = self.conn.state.lock_for_drop() else { + let Some(mut conn) = self.stream.conn.state.lock_for_drop() else { return; }; - - // clean up any previously registered wakers - conn.blocked_writers.remove(&self.id); - - if conn.error.is_none() { - let mut stream = conn.inner.send_stream(self.id); - let stopped = match stream.end() { - Ok(Some(End::Stopped(code))) => Some(code), - _ => None, - }; - let reset = match (&self.state, stopped) { - (HalfState::Reset, _) => None, - // Echo the code, as RFC 9000 section 3.5 asks, even after `finish`: the peer reads - // nothing more, so retransmitting the rest is wasted. - (_, Some(code)) => Some(stream.reset(code)), - (HalfState::Finished, None) => None, - (HalfState::Open(offset, code), None) => Some(match offset { - Some(offset) => stream.reset_at(offset.inner, *code), - None => stream.reset(*code), - }), - }; - if reset.is_some_and(|reset| reset.is_ok()) { - conn.resolve_stopped(self.id, stopped); - conn.wake(); - } + if conn.is_lost() { + return; + } + let mut stream = conn.inner.send_stream(self.stream.id); + let stopped = match stream.end() { + Ok(Some(End::Stopped(code))) => Some(code), + _ => None, + }; + let reset = match (&self.state, stopped) { + (HalfState::Reset, _) => None, + // Echo the code, as RFC 9000 section 3.5 asks, even after `finish`: the peer reads + // nothing more, so retransmitting the rest is wasted. + (_, Some(code)) => Some(stream.reset(code)), + (HalfState::Finished, None) => None, + (HalfState::Open(offset, code), None) => Some(match offset { + Some(offset) => stream.reset_at(offset.inner, *code), + None => stream.reset(*code), + }), + }; + if reset.is_some_and(|reset| reset.is_ok()) { + let _ = self.stream.reset.set(stopped); } - conn.inner.send_stream(self.id).release(); + // `stream` releases the ID once this and every `stopped` future dropped it, after the + // lock here is let go. } } impl tokio::io::AsyncWrite for Stream { fn poll_write(self: Pin<&mut Self>, cx: &mut Context<'_>, mut buf: &[u8]) -> Poll> { let this = self.get_mut(); - if !matches!(this.0.state, HalfState::Open(..)) { + if !matches!(this.half.state, HalfState::Open(..)) { return Poll::Ready(Err(io::ErrorKind::BrokenPipe.into())); } - this.poll_write(cx, &mut buf).map_err(Into::into) + let waiter = this.park.hold(cx); + this.half.poll_write(waiter, &mut buf).map_err(Into::into) } fn poll_flush(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll> { @@ -420,8 +412,8 @@ impl tokio::io::AsyncWrite for Stream { fn poll_shutdown(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll> { let this = self.get_mut(); - if matches!(this.0.state, HalfState::Open(..)) { - this.0.finish(); + if matches!(this.half.state, HalfState::Open(..)) { + this.half.finish(); } Poll::Ready(Ok(())) } diff --git a/rs/moq-tokio/src/quic/driver/tests.rs b/rs/moq-tokio/src/quic/driver/tests.rs index 2e3ced56f6c..658d2adc254 100755 --- a/rs/moq-tokio/src/quic/driver/tests.rs +++ b/rs/moq-tokio/src/quic/driver/tests.rs @@ -1850,14 +1850,16 @@ async fn dropped_stream_resolves_stopped() { let (conn, _server) = connect_self(&endpoint).await; let mut send = conn.open_uni(Default::default()).await.unwrap(); send.write_all(&b"truncated"[..]).await.unwrap(); + let id = send.id(); let stopped = send.stopped(); drop(send); + assert!(!conn.send_released(id), "the stopped future keeps the stream's end"); let stopped = timeout(Duration::from_secs(10), stopped).await; assert_eq!(stopped, Ok(Ok(None))); - assert_eq!(conn.stopped_waiters(), 0); + assert!(conn.send_released(id), "the last stopped future releases the stream"); } -/// A reset releases the stream's `stopped` waiters, even one dropped while pending +/// A `stopped` future dropped while pending leaves the stream to release itself #[tokio::test] async fn stream_reset_releases_stopped() { let _guard = subscribe(); @@ -1865,18 +1867,21 @@ async fn stream_reset_releases_stopped() { let (conn, _server) = connect_self(&endpoint).await; let mut send = conn.open_uni(Default::default()).await.unwrap(); send.write_all(&b"hello"[..]).await.unwrap(); + let id = send.id(); { let mut stopped = pin!(send.stopped()); let poll = std::future::poll_fn(|cx| Poll::Ready(stopped.as_mut().poll(cx))).await; assert!(poll.is_pending()); } - assert_eq!(conn.stopped_waiters(), 1); + assert!(!conn.send_released(id)); send.reset(7u32.into()); - assert_eq!(conn.stopped_waiters(), 0); + assert!(conn.send_released(id)); } +/// A blocked read wakes when the connection closes, and one dropped with its stream leaves no +/// waiter behind #[tokio::test] -async fn stream_drop_removes_blocked_reader() { +async fn dropped_read_leaves_no_waiter() { let _guard = subscribe(); for drop_stream in [false, true] { @@ -1895,24 +1900,20 @@ async fn stream_drop_removes_blocked_reader() { let (waker, wake_counter) = new_count_waker(); let mut cx = Context::from_waker(&waker); - // do a blocking read which will add the stream in conn.blocked_readers - { - let mut buf = [0u8; 64]; - let mut buf = &mut buf[..]; - let read_fut = stream.read(&mut buf); - tokio::pin!(read_fut); - assert!(matches!(read_fut.as_mut().poll(&mut cx), Poll::Pending)); - } + // A blocking read registers its waiter with moq-quic. + let mut buf = [0u8; 64]; + let mut buf = &mut buf[..]; + let mut read_fut = Box::pin(stream.read(&mut buf)); + assert!(matches!(read_fut.as_mut().poll(&mut cx), Poll::Pending)); if !drop_stream { assert_eq!(wake_counter.wakes(), 0); - // We have a blocked reader, closing the connection should wake it. We use this as - // a proxy to assert that the stream is in conn.blocked_readers. + // Closing the connection wakes the blocked reader. conn.close(0u32.into(), "done"); assert_eq!(wake_counter.wakes(), 1); } else { - // dropping the stream should remove it from conn.blocked_readers, so we don't - // expect any wakeups + // The dropped read took its waiter with it, so nothing is left to wake. + drop(read_fut); drop(stream); assert_eq!(wake_counter.wakes(), 0, "no wakeups should have occurred"); conn.close(0u32.into(), "done"); diff --git a/rs/moq-uring/src/quic/driver/connection.rs b/rs/moq-uring/src/quic/driver/connection.rs index 2b67da3cb5d..32f7cd36d16 100644 --- a/rs/moq-uring/src/quic/driver/connection.rs +++ b/rs/moq-uring/src/quic/driver/connection.rs @@ -4,12 +4,11 @@ use kio::clock::Instant; use std::cell::{Cell, RefCell}; use std::net::SocketAddr; use std::rc::{Rc, Weak}; -use std::task::{Context, Poll}; +use std::task::{Context, Poll, ready}; use bytes::{Buf, Bytes}; use moq_quic::endpoint::ConnectionHandle; -use moq_quic::stream::{Dir, Id as StreamId}; -use rustc_hash::FxHashMap; +use moq_quic::stream::Dir; use super::super::{Error, VarInt}; use super::{endpoint, recv, send}; @@ -33,8 +32,10 @@ const TRAINS_PER_TURN: usize = 4; /// Everything the handles and the driver share, single-threaded. /// /// The connection and the bookkeeping live in separate `RefCell`s so a stream -/// operation can mutate the connection, drop that borrow, and then register a -/// waiter without ever holding both. +/// operation can mutate the connection, drop that borrow, and then read the +/// terminal error without ever holding both. moq-quic owns every handle's wait +/// and the driver's transmit wait; the bookkeeping is the driver's ingress wake +/// and the close. pub(crate) struct Inner { pub(crate) conn: RefCell, /// Updated by packet ingress and egress, so readers never wake the worker to sample. @@ -53,34 +54,22 @@ pub(crate) struct Inner { pub(crate) struct State { pub(crate) active: Option, - /// Wakes the driver; handles kick it after mutating the connection so - /// fresh egress reaches the wire. + /// Wakes the driver when the endpoint fed the connection (a received + /// datagram, an unreachable peer) so it answers and re-arms its timers, or + /// failed it from outside. Handles need no wake of their own: whatever + /// they queue wakes the driver's `poll_transmit`. driver: kio::WaiterList, - established: bool, - establish_waiters: kio::WaiterList, - - /// Whoever is waiting for a peer-initiated stream. moq-quic hands the - /// ids out in order, so the queue is its own. - accept_bi_waiters: kio::WaiterList, - accept_uni_waiters: kio::WaiterList, - - /// Whoever is blocked on the peer's MAX_STREAMS credit. - open_waiters: kio::WaiterList, - - /// Per-stream read/write parking, keyed by stream id. + /// The terminal error as the handle's `poll_closed` reports it, set exactly + /// once: moq-quic's reason, published by the driver (a local close only once + /// staged, see `Driver::publish_close`), or the driver's own failure, which + /// moq-quic cannot take and so overrides its reason everywhere. /// - /// FxHash rather than SipHash on all four of these: they sit on the - /// driver's per-event path, and moq-quic assigns the ids. - readable: FxHashMap, - writable: FxHashMap, - /// Send streams waiting for their end: a FIN the peer acknowledged, or a - /// `STOP_SENDING`. - finishing: FxHashMap, - - datagram_recv_waiters: kio::WaiterList, - - /// The terminal error, set exactly once; everything fails with it after. + /// A latch of its own rather than moq-quic's `poll_closed`, which resolves a + /// local close as soon as it is asked for. Only the driver knows when the + /// CONNECTION_CLOSE is staged on the socket, which moq-quic cannot see, and + /// a caller may stop driving the worker the moment `poll_closed` resolves, + /// so publishing earlier would leave the packet unsent. closed: Option, closed_waiters: kio::WaiterList, @@ -98,15 +87,6 @@ impl State { Self { active: None, driver: kio::WaiterList::new(), - established: false, - establish_waiters: kio::WaiterList::new(), - accept_bi_waiters: kio::WaiterList::new(), - accept_uni_waiters: kio::WaiterList::new(), - open_waiters: kio::WaiterList::new(), - readable: FxHashMap::default(), - writable: FxHashMap::default(), - finishing: FxHashMap::default(), - datagram_recv_waiters: kio::WaiterList::new(), closed: None, closed_waiters: kio::WaiterList::new(), dead: false, @@ -114,21 +94,8 @@ impl State { } } - /// Drop send stream `id`'s bookkeeping, parking included. - fn forget_send(&mut self, id: StreamId) { - self.finishing.remove(&id); - self.writable.remove(&id); - } - - /// The same for the read half. Only its own table: the two halves of a - /// bidirectional stream are separate handles, and the other one may still - /// be parked. - fn forget_recv(&mut self, id: StreamId) { - self.readable.remove(&id); - } - - /// Terminate with `err` (the first one wins) and wake absolutely everyone, - /// the driver included: an externally failed driver has to notice. + /// Terminate with `err` (the first one wins) and wake the closed waiters and + /// the driver: an externally failed driver has to notice. fn fail(&mut self, err: Error) { self.active.take(); if self.closed.is_none() { @@ -136,27 +103,25 @@ impl State { } self.driver.wake(); self.closed_waiters.wake(); - self.establish_waiters.wake(); - self.accept_bi_waiters.wake(); - self.accept_uni_waiters.wake(); - self.open_waiters.wake(); - self.datagram_recv_waiters.wake(); - for waiters in self.readable.values_mut() { - waiters.wake(); - } - for waiters in self.writable.values_mut() { - waiters.wake(); - } - for waiters in self.finishing.values_mut() { - waiters.wake(); - } } } impl Inner { - /// The terminal error, if the connection has one. + /// The terminal error, if the connection has one: the driver's own failure, + /// which wins, or why moq-quic lost it. + /// + /// Unlike the handle's `poll_closed`, a local close counts from the moment it + /// was asked for, as moq-quic's stream waits already fail by then. pub(crate) fn closed(&self) -> Option { - self.state.borrow().closed.clone() + if let Some(err) = &self.state.borrow().closed { + return Some(err.clone()); + } + self.conn.borrow().error().cloned().map(Error::from) + } + + /// The terminal error, for a wait moq-quic failed with `ConnectionLost`. + pub(crate) fn lost(&self) -> Error { + self.closed().expect("moq-quic reported the connection lost") } /// Count a new connection or stream handle. @@ -174,50 +139,27 @@ impl Inner { } } - /// Wake the driver so it flushes what a handle just queued. + /// Wake the driver after the endpoint fed the connection, so it answers and + /// re-arms its timers. pub(crate) fn kick(&self) { self.state.borrow_mut().driver.wake(); } - /// Terminate from outside (the endpoint's socket died): everything fails - /// with `err`, and the driver exits on its next poll. - pub(crate) fn fail(&self, err: Error) { - let mut state = self.state.borrow_mut(); - state.dead = true; - state.fail(err); - } - - /// Park `waiter` until stream `id` is writable again. - pub(crate) fn park_writable(&self, id: StreamId, waiter: &kio::Waiter) { - let mut state = self.state.borrow_mut(); - waiter.register(state.writable.entry(id).or_default()); - } - - /// Park `waiter` until stream `id` is readable again. - pub(crate) fn park_readable(&self, id: StreamId, waiter: &kio::Waiter) { - let mut state = self.state.borrow_mut(); - waiter.register(state.readable.entry(id).or_default()); - } - - /// Park `waiter` until send stream `id` reaches its end. - pub(crate) fn park_finishing(&self, id: StreamId, waiter: &kio::Waiter) { - let mut state = self.state.borrow_mut(); - waiter.register(state.finishing.entry(id).or_default()); - } - - /// Forget send stream `id`'s bookkeeping; called when its handle drops. + /// Terminate from outside (the socket died): everything fails with `err`, + /// and the driver exits on its next poll. /// - /// The parking table goes with it. A stream reset on the way out is never - /// reported writable again, so nothing else would ever remove the entry, - /// and a peer withholding flow control credit could have streams opened - /// and cancelled against it without bound. - pub(crate) fn forget_send(&self, id: StreamId) { - self.state.borrow_mut().forget_send(id); - } - - /// The same for the read half, which parks on its own table. - pub(crate) fn forget_recv(&self, id: StreamId) { - self.state.borrow_mut().forget_recv(id); + /// moq-quic takes no foreign error, so it is closed as well, which wakes + /// every handle parked in it; they then report `err`, which wins. + pub(crate) fn fail(&self, err: Error) { + let reason = err.to_string(); + { + let mut state = self.state.borrow_mut(); + state.dead = true; + state.fail(err); + } + self.conn + .borrow_mut() + .close(self.owner.now(), VarInt::from_u32(0), reason); } /// Close the connection with a full-width application code. @@ -226,12 +168,13 @@ impl Inner { /// to `u32`; the WebTransport layer maps its codes into HTTP/3's error /// space, which needs the whole varint range. pub(crate) fn close_code(&self, code: u64, reason: &str) { + // The close wakes the driver itself, through its `poll_closed` and + // `poll_transmit` waits, even one stalled on a full transmit pool. self.conn.borrow_mut().close( self.owner.now(), VarInt::from_u64(code).unwrap_or(VarInt::MAX), reason.to_string(), ); - self.state.borrow_mut().driver.wake(); } } @@ -306,7 +249,6 @@ impl Connection { Err(moq_quic::send::Error::TooLarge) => return Err(send::Error::TooLarge), Err(_) => return Err(send::Error::Unsupported), } - self.shared.kick(); Ok(()) } @@ -377,7 +319,6 @@ impl Connection { /// refuses. pub fn set_limits(&self, limits: crate::quic::Limits) -> Result<(), Error> { self.shared.conn.borrow_mut().set_limits(limits)?; - self.shared.kick(); Ok(()) } @@ -419,7 +360,7 @@ impl Clone for Connection { /// Build a connection's shared state and its driver. /// -/// The driver future does timers, event sweeps, and egress; ingress arrives +/// The driver future does timers, the close, and egress; ingress arrives /// from the endpoint's demux task, which feeds moq-quic directly and /// [kicks](Inner::kick) the driver. The caller spawns the future and reclaims /// the connection's bookkeeping once it resolves. @@ -496,15 +437,12 @@ pub(crate) async fn establish(shared: Shared, deadline: Option) -> Resu expired = true; shared.close_code(0, "QUIC handshake timed out"); } - let mut state = shared.state.borrow_mut(); - if state.established && !expired { + let connected = shared.conn.borrow_mut().poll_connected(waiter); + if !expired && matches!(connected, Poll::Ready(Ok(()))) { return Poll::Ready(Ok(())); } - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - waiter.register(&mut state.establish_waiters); - Poll::Pending + // A failed or refused handshake resolves with the published close. + poll_closed(&shared, waiter).map(Err) }) .await; if result.is_err() { @@ -659,31 +597,22 @@ impl Drop for Connection { } fn accept_uni(shared: &Shared, waiter: &kio::Waiter) -> Poll> { - if let Some(id) = shared.conn.borrow_mut().streams().accept(Dir::Uni) { - return Poll::Ready(Ok(recv::Stream::new(shared.clone(), id))); + let accepted = shared.conn.borrow_mut().streams().poll_accept(waiter, Dir::Uni); + match ready!(accepted) { + Ok(id) => Poll::Ready(Ok(recv::Stream::new(shared.clone(), id))), + Err(_) => Poll::Ready(Err(shared.lost())), } - let mut state = shared.state.borrow_mut(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - waiter.register(&mut state.accept_uni_waiters); - Poll::Pending } fn accept_bi(shared: &Shared, waiter: &kio::Waiter) -> Poll> { - let accepted = shared.conn.borrow_mut().streams().accept(Dir::Bi); - if let Some(id) = accepted { - return Poll::Ready(Ok(( + let accepted = shared.conn.borrow_mut().streams().poll_accept(waiter, Dir::Bi); + match ready!(accepted) { + Ok(id) => Poll::Ready(Ok(( send::Stream::new(shared.clone(), id), recv::Stream::new(shared.clone(), id), - ))); - } - let mut state = shared.state.borrow_mut(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); + ))), + Err(_) => Poll::Ready(Err(shared.lost())), } - waiter.register(&mut state.accept_bi_waiters); - Poll::Pending } /// Open a stream in `dir`, with its receive half when bidirectional. @@ -693,32 +622,29 @@ fn open( options: send::Options<'_>, waiter: &kio::Waiter, ) -> Poll), Error>> { - if let Some(err) = shared.closed() { - return Poll::Ready(Err(err)); - } - // The peer's MAX_STREAMS credit is what gates us: `open` hands back nothing while it is - // spent. - let opened = shared.conn.borrow_mut().streams().open(dir, options.to_quic()); - let Some(id) = opened else { - waiter.register(&mut shared.state.borrow_mut().open_waiters); - return Poll::Pending; + // The peer's MAX_STREAMS credit is what gates us, once the connection is not lost. + let opened = shared + .conn + .borrow_mut() + .streams() + .poll_open(waiter, dir, options.to_quic()); + let id = match ready!(opened) { + Ok(id) => id, + Err(_) => return Poll::Ready(Err(shared.lost())), }; let recv = (dir == Dir::Bi).then(|| recv::Stream::new(shared.clone(), id)); Poll::Ready(Ok((send::Stream::new(shared.clone(), id), recv))) } fn recv_datagram(shared: &Shared, waiter: &kio::Waiter) -> Poll> { - if let Some(datagram) = shared.conn.borrow_mut().datagrams().recv() { - return Poll::Ready(Ok(datagram)); - } - let mut state = shared.state.borrow_mut(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); + let received = shared.conn.borrow_mut().datagrams().poll_recv(waiter); + match ready!(received) { + Ok(datagram) => Poll::Ready(Ok(datagram)), + Err(_) => Poll::Ready(Err(shared.lost())), } - waiter.register(&mut state.datagram_recv_waiters); - Poll::Pending } +/// Wait for the published terminal error, which a local close reaches only once staged. fn poll_closed(shared: &Shared, waiter: &kio::Waiter) -> Poll { let mut state = shared.state.borrow_mut(); if let Some(err) = &state.closed { @@ -790,9 +716,10 @@ impl web_transport_trait::Stats for Stats { } } -/// The per-connection task: endpoint events, application events, timers, and -/// packets out. Packets in come from the endpoint's demux task, which feeds -/// moq-quic directly and kicks this driver. +/// The per-connection task: endpoint events, the connection's loss, timers, +/// and packets out. Packets in come from the endpoint's demux task, which feeds +/// moq-quic directly and kicks this driver; what the handles queue wakes it +/// through `poll_transmit`. struct Driver { shared: Shared, socket: Rc, @@ -815,11 +742,14 @@ struct Driver { /// The worker dropped its tasks, this driver among them: nothing will ever /// drive the connection again, so every handle fails rather than pending. /// After a normal end the terminal error is already set, and this keeps it. +/// +/// Failing from outside closes moq-quic too, which wakes every handle parked +/// in its waits. impl Drop for Driver { fn drop(&mut self) { - let mut state = self.shared.state.borrow_mut(); - if state.closed.is_none() { - state.fail(Error::Io(crate::shared::Shared::gone_error().to_string())); + if self.shared.state.borrow().closed.is_none() { + self.shared + .fail(Error::Io(crate::shared::Shared::gone_error().to_string())); } } } @@ -833,14 +763,17 @@ impl Driver { if state.dead { return Poll::Ready(()); } - // Register the kick first: a handle mutating the connection after - // this turn's sweeps still re-polls us. + // Register for ingress first: a datagram the endpoint feeds the + // connection after this turn's checks still re-polls us. waiter.register(&mut state.driver); } loop { self.endpoint_events(); - self.sweep(); + // Handles wait on moq-quic itself, so its events go unread; drain them + // so the queue stays empty. + while self.shared.conn.borrow_mut().poll().is_some() {} + self.check_lost(waiter); if self.shared.state.borrow().dead { return Poll::Ready(()); @@ -856,7 +789,7 @@ impl Driver { } if let Poll::Ready(err) = self.flush(waiter) { - self.shared.state.borrow_mut().fail(err); + self.shared.fail(err); return Poll::Ready(()); } if self.close_staged { @@ -895,34 +828,23 @@ impl Driver { } } - /// Everything event-shaped: establishment, new and readable streams, - /// writability, finished sends, received datagrams, and the end. - fn sweep(&mut self) { - loop { - let event = self.shared.conn.borrow_mut().poll(); - let Some(event) = event else { - return; - }; - - let mut state = self.shared.state.borrow_mut(); - match event { - moq_quic::Event::Connected => { - state.established = true; - state.establish_waiters.wake(); - } - // The driver publishes a local close once it has staged the - // CONNECTION_CLOSE, so a caller that stops driving the worker the - // moment `poll_closed` resolves has at least handed the packet over. - moq_quic::Event::ConnectionLost { - reason: reason @ moq_quic::connection::Error::LocallyClosed { .. }, - } => state.local_close = Some(reason.into()), - moq_quic::Event::ConnectionLost { reason } => state.fail(reason.into()), - moq_quic::Event::DatagramReceived => state.datagram_recv_waiters.wake(), - moq_quic::Event::HandshakeDataReady => {} - moq_quic::Event::Stream(event) => sweep_stream(&mut state, event), - // Non-exhaustive upstream; nothing else needs a wakeup. - _ => {} - } + /// Learn whether moq-quic lost the connection, and publish why. + /// + /// The driver publishes a local close once it has staged the + /// CONNECTION_CLOSE, so a caller that stops driving the worker the moment + /// `poll_closed` resolves has at least handed the packet over. + fn check_lost(&mut self, waiter: &kio::Waiter) { + let lost = self.shared.conn.borrow_mut().poll_closed(waiter); + let Poll::Ready(reason) = lost else { + return; + }; + let mut state = self.shared.state.borrow_mut(); + if state.closed.is_some() || state.local_close.is_some() { + return; + } + match reason { + reason @ moq_quic::connection::Error::LocallyClosed { .. } => state.local_close = Some(reason.into()), + reason => state.fail(reason.into()), } } @@ -966,9 +888,10 @@ impl Driver { /// Fill one transmit buffer and stage it. /// - /// Pacing happens inside `poll_transmit`: a paced path returns nothing - /// and arms a timer that `poll_timeout` reports, and the driver loop - /// flushes again once it fires. + /// `poll_transmit` is the driver's wait for egress: with nothing to send it + /// registers `waiter`, which whatever a handle queues wakes. Pacing happens + /// inside it too: a paced path returns `Pending` and arms a timer that + /// `poll_timeout` reports, and the driver loop flushes again once it fires. fn flush_one(&mut self, waiter: &kio::Waiter) -> Poll> { let mut tx = match self.socket.poll_acquire(waiter) { Poll::Ready(Ok(tx)) => tx, @@ -991,13 +914,15 @@ impl Driver { )))); } // A new target waits for the next egress: probing only resends data in flight, so it - // has nothing to send until a write or an ACK kicks the driver anyway. + // has nothing to send until a write or an ACK wakes the driver anyway. if let Some(probe) = &self.shared.probe { conn.set_probe_target(probe.target().as_bps() / 8); } // moq-quic encrypts the train straight into the socket's buffer. - // Nothing to send, or paced; the buffer returns to the pool on drop. - let Some(transmit) = conn.poll_transmit(self.shared.owner.now(), segments, &mut tx) else { + // Nothing to send registers us until the application queues + // something; paced, the timer re-polls us. Either way the buffer + // returns to the pool on drop. + let Poll::Ready(transmit) = conn.poll_transmit(waiter, self.shared.owner.now(), segments, &mut tx) else { return Poll::Pending; }; self.shared.wire.set_sent_bytes(conn.stats().udp_tx.bytes); @@ -1021,71 +946,3 @@ impl Driver { Poll::Ready(Ok(())) } } - -/// Wake whoever waits for send stream `id` to end. -fn ended(state: &mut State, id: StreamId) { - if let Some(mut waiters) = state.finishing.remove(&id) { - waiters.wake(); - } -} - -/// Apply one stream event to the parking tables. -fn sweep_stream(state: &mut State, event: moq_quic::StreamEvent) { - // Waking removes the entry: a still-interested poller re-registers on its - // next poll, so the maps only hold streams somebody is parked on. - match event { - moq_quic::StreamEvent::Opened { dir: Dir::Bi } => state.accept_bi_waiters.wake(), - moq_quic::StreamEvent::Opened { dir: Dir::Uni } => state.accept_uni_waiters.wake(), - moq_quic::StreamEvent::Available { .. } => state.open_waiters.wake(), - moq_quic::StreamEvent::Readable { id } => { - if let Some(mut waiters) = state.readable.remove(&id) { - waiters.wake(); - } - } - moq_quic::StreamEvent::Writable { id } => { - if let Some(mut waiters) = state.writable.remove(&id) { - waiters.wake(); - } - } - moq_quic::StreamEvent::Finished { id } => ended(state, id), - moq_quic::StreamEvent::Stopped { id, .. } => { - ended(state, id); - // A writer blocked on capacity has to learn it will never come. - if let Some(mut waiters) = state.writable.remove(&id) { - waiters.wake(); - } - } - // Nothing waits on a send stream's acknowledgment yet. - moq_quic::StreamEvent::Acked { .. } => {} - } -} - -#[cfg(test)] -mod tests { - use moq_quic::Side; - - use super::*; - - /// A dropped handle takes its parking with it. - /// - /// Nothing else can: a stream reset or stopped on the way out is never - /// reported writable or readable again, so an entry left behind sits - /// there for the life of the connection, one per cancelled stream. - #[test] - fn forgetting_a_handle_clears_its_parking() { - let mut state = State::new(); - let id = StreamId::new(Side::Client, Dir::Bi, 0); - state.writable.entry(id).or_default(); - state.readable.entry(id).or_default(); - state.finishing.entry(id).or_default(); - - state.forget_send(id); - assert!(state.writable.is_empty(), "the write half's parking"); - assert!(state.finishing.is_empty(), "the finish watch"); - // The read half is a separate handle, which may still be parked. - assert!(!state.readable.is_empty(), "the read half's parking survives"); - - state.forget_recv(id); - assert!(state.readable.is_empty(), "the read half's parking"); - } -} diff --git a/rs/moq-uring/src/quic/driver/recv.rs b/rs/moq-uring/src/quic/driver/recv.rs index b3c0f905362..f4a72fe8598 100644 --- a/rs/moq-uring/src/quic/driver/recv.rs +++ b/rs/moq-uring/src/quic/driver/recv.rs @@ -1,7 +1,8 @@ //! Incoming streams: their handles, leases, and errors. //! -//! A read comes straight out of moq-quic's reassembly buffer; the handles kick the driver when -//! a read owes the peer credit and park on the per-stream waiter lists the driver wakes. +//! A read comes straight out of moq-quic's reassembly buffer, which wakes the driver's transmit +//! wait itself when a read owes the peer credit, and the handles wait through moq-quic's poll +//! forms, which wake them. use std::future::poll_fn; use std::task::{Context, Poll, ready}; @@ -55,7 +56,7 @@ impl Stream { return Poll::Ready(Ok(Some(0))); } self.parked(cx, |this, waiter| { - this.poll_read_generic(waiter, |chunks| fill(chunks, buf)) + this.poll_read_generic(waiter, |chunks, waiter| fill(chunks, waiter, buf)) }) } @@ -86,9 +87,10 @@ impl Stream { /// [`Self::read_chunk`], as a poll. pub(crate) fn poll_read_chunk(&mut self, cx: &Context<'_>, max: usize) -> Poll, Error>> { self.parked(cx, |this, waiter| { - this.poll_read_generic(waiter, |chunks| match chunks.next(max) { - Ok(Some(chunk)) => ReadStatus::Readable(chunk.bytes), - res => (None, res.err()).into(), + this.poll_read_generic(waiter, |chunks, waiter| match chunks.poll_next(waiter, max) { + Poll::Ready(Ok(Some(chunk))) => ReadStatus::Readable(chunk.bytes), + Poll::Ready(res) => (None, res.err()).into(), + Poll::Pending => ReadStatus::Failed(None, moq_quic::recv::Error::Blocked), }) }) } @@ -159,19 +161,14 @@ impl Stream { if let Some(code) = self.reset { return Poll::Ready(Err(Error::Reset(code))); } - let (result, transmit) = self.shared.conn.borrow_mut().recv_stream(self.id).lease(want); - if transmit.should_transmit() { - self.shared.kick(); - } - let err = match result { + let (result, _) = self + .shared + .conn + .borrow_mut() + .recv_stream(self.id) + .poll_lease(waiter, want); + let err = match ready!(result) { Ok(lease) => return Poll::Ready(Ok(lease)), - Err(moq_quic::recv::Error::Blocked) => { - if let Some(err) = self.shared.closed() { - return Poll::Ready(Err(Error::Connection(err))); - } - self.shared.park_readable(self.id, waiter); - return Poll::Pending; - } Err(moq_quic::recv::Error::Reset(code)) => Error::Reset(code), Err(moq_quic::recv::Error::Evicted) => Error::Evicted, Err(moq_quic::recv::Error::TooLarge) => Error::TooLarge, @@ -229,26 +226,22 @@ impl Stream { if let Some(code) = self.reset { return Poll::Ready(Ok(Some(code))); } - let reset = self.shared.conn.borrow_mut().recv_stream(self.id).received_reset(); - match reset { + let reset = self + .shared + .conn + .borrow_mut() + .recv_stream(self.id) + .poll_received_reset(waiter); + match ready!(reset) { // Evicted, so its bytes are gone: nothing frees the window until it is stopped. Err(moq_quic::recv::Error::Evicted) => Poll::Ready(Err(Error::Evicted)), - Err(_) => Poll::Ready(Ok(None)), - Ok(Some(code)) => { - // The stream's state was just freed, so the peer may be owed stream credit, and - // later reads fail with the cached code. + Ok(code) => { + // The stream's state was just freed, so later reads fail with the cached code. self.reset = Some(code); - self.shared.kick(); Poll::Ready(Ok(Some(code))) } - Ok(None) => match self.shared.closed() { - Some(err) => Poll::Ready(Err(Error::Connection(err))), - // A reset is a read error, so it always wakes readers. - None => { - self.shared.park_readable(self.id, waiter); - Poll::Pending - } - }, + Err(moq_quic::recv::Error::ConnectionLost) => Poll::Ready(Err(Error::Connection(self.shared.lost()))), + Err(_) => Poll::Ready(Ok(None)), } } @@ -262,29 +255,33 @@ impl Stream { if let Some(code) = self.reset { return Poll::Ready(Ok(Some(code))); } - let read = self.parked(cx, |this, waiter| { - this.poll_read_generic(waiter, |chunks| match chunks.next(0) { - // Bytes are still unread, so the end is not reached. - Ok(Some(_)) => ReadStatus::Failed(None, moq_quic::recv::Error::Blocked), - res => (None::<()>, res.err()).into(), - }) - }); - Poll::Ready(match ready!(read) { - Ok(_) => Ok(None), - Err(Error::Reset(code)) => Ok(Some(code)), - Err(err) => Err(err), + self.parked(cx, |this, waiter| { + let read = this.poll_read_generic(waiter, |chunks, waiter| match chunks.poll_next(waiter, 0) { + Poll::Ready(Ok(Some(_))) => ReadStatus::Readable(()), + Poll::Ready(res) => (None, res.err()).into(), + Poll::Pending => ReadStatus::Failed(None, moq_quic::recv::Error::Blocked), + }); + match ready!(read) { + // Bytes are still unread, so the end is not reached: wait for a reset, or for + // more to arrive, as a reader would. + Ok(Some(())) => this.wait_received_reset(waiter), + Ok(None) => Poll::Ready(Ok(None)), + Err(Error::Reset(code)) => Poll::Ready(Ok(Some(code))), + Err(err) => Poll::Ready(Err(err)), + } }) } /// Handle the logic common to every read out of a receive stream. /// - /// `read_fn` reads from the passed `Chunks` and returns the status after reading: the - /// amount of data read, and the status after the final read call. + /// `read_fn` reads from the passed `Chunks`, polling with the waiter, and returns the + /// status after reading: the amount of data read, and the status after the final read call, + /// [`Blocked`](moq_quic::recv::Error::Blocked) once the waiter is registered. fn poll_read_generic(&mut self, waiter: &kio::Waiter, mut read_fn: T) -> Poll, Error>> where - T: FnMut(&mut Chunks<'_>) -> ReadStatus, + T: FnMut(&mut Chunks<'_>, &kio::Waiter) -> ReadStatus, { - use moq_quic::recv::Error::{Blocked, Reset}; + use moq_quic::recv::Error::{Blocked, ConnectionLost, Reset}; if self.all_data_read { return Poll::Ready(Ok(None)); @@ -295,19 +292,15 @@ impl Stream { Some(code) => ReadStatus::Failed(None, Reset(code)), None => { let mut conn = self.shared.conn.borrow_mut(); - let (status, transmit) = match conn.recv_stream(self.id).read() { + match conn.recv_stream(self.id).read() { Ok(mut chunks) => { - let status = read_fn(&mut chunks); - (status, chunks.finalize().should_transmit()) + let status = read_fn(&mut chunks, waiter); + // Reading is what returns the peer's flow control credit. + let _ = chunks.finalize(); + status } - Err(err) => (ReadStatus::Failed(None, err), false), - }; - drop(conn); - // Reading is what returns the peer's flow control credit. - if transmit { - self.shared.kick(); + Err(err) => ReadStatus::Failed(None, err), } - status } }; @@ -317,14 +310,10 @@ impl Stream { self.all_data_read = true; Poll::Ready(Ok(read)) } - ReadStatus::Failed(Some(read), Blocked) => Poll::Ready(Ok(Some(read))), - ReadStatus::Failed(None, Blocked) => { - if let Some(err) = self.shared.closed() { - return Poll::Ready(Err(Error::Connection(err))); - } - self.shared.park_readable(self.id, waiter); - Poll::Pending - } + // What was read is returned before the loss is reported. + ReadStatus::Failed(Some(read), Blocked | ConnectionLost) => Poll::Ready(Ok(Some(read))), + // moq-quic wakes the waiter once more arrives. + ReadStatus::Failed(None, Blocked) => Poll::Pending, ReadStatus::Failed(read, Reset(code)) => { self.reset = Some(code); if read.is_some() { @@ -348,7 +337,9 @@ impl Stream { } /// Copy from `chunks` until `buf` is full or nothing more is readable, returning how much. -fn fill(chunks: &mut Chunks<'_>, buf: &mut impl BufMut) -> ReadStatus { +/// +/// Nothing more readable registers `waiter`, which is harmless once something was read. +fn fill(chunks: &mut Chunks<'_>, waiter: &kio::Waiter, buf: &mut impl BufMut) -> ReadStatus { let mut read = 0; loop { if !buf.has_remaining_mut() { @@ -356,23 +347,23 @@ fn fill(chunks: &mut Chunks<'_>, buf: &mut impl BufMut) -> ReadStatus { return ReadStatus::Readable(read); } - match chunks.next(buf.remaining_mut()) { - Ok(Some(chunk)) => { + let read_some = (read > 0).then_some(read); + match chunks.poll_next(waiter, buf.remaining_mut()) { + Poll::Ready(Ok(Some(chunk))) => { buf.put_slice(&chunk.bytes); read += chunk.bytes.len(); } - res => return ((read > 0).then_some(read), res.err()).into(), + Poll::Ready(res) => return (read_some, res.err()).into(), + Poll::Pending => return ReadStatus::Failed(read_some, moq_quic::recv::Error::Blocked), } } } impl Drop for Stream { fn drop(&mut self) { - self.shared.forget_recv(self.id); if !self.all_data_read && self.shared.closed().is_none() { // Fails harmlessly once the stream is gone. let _ = self.shared.conn.borrow_mut().recv_stream(self.id).stop(self.drop_stop); - self.shared.kick(); } self.shared.release(); } @@ -419,10 +410,7 @@ impl Lease<'_> { pub fn release(mut self) { let lease = self.inner.take().expect("only taken to hand it back"); let stream = &*self.stream; - let transmit = stream.shared.conn.borrow_mut().recv_stream(stream.id).release(lease); - if transmit.should_transmit() { - stream.shared.kick(); - } + let _ = stream.shared.conn.borrow_mut().recv_stream(stream.id).release(lease); } } @@ -449,10 +437,7 @@ impl Drop for Lease<'_> { fn drop(&mut self) { if let Some(lease) = self.inner.take() { let stream = &*self.stream; - let transmit = stream.shared.conn.borrow_mut().recv_stream(stream.id).restore(lease); - if transmit.should_transmit() { - stream.shared.kick(); - } + let _ = stream.shared.conn.borrow_mut().recv_stream(stream.id).restore(lease); } } } diff --git a/rs/moq-uring/src/quic/driver/send.rs b/rs/moq-uring/src/quic/driver/send.rs index 7a0760d2a16..0485329712d 100644 --- a/rs/moq-uring/src/quic/driver/send.rs +++ b/rs/moq-uring/src/quic/driver/send.rs @@ -1,12 +1,12 @@ //! Outgoing streams and datagrams: their handles, scheduling, and errors. //! //! Single-threaded sans-IO means a write goes straight into moq-quic's send queue (no staging -//! copy); the handles just kick the driver so egress reaches the wire and park on the -//! per-stream waiter lists the driver wakes. +//! copy), which wakes the driver's transmit wait itself, and the handles wait through +//! moq-quic's poll forms, which wake them. use std::future::poll_fn; use std::rc::Rc; -use std::task::{Context, Poll}; +use std::task::{Context, Poll, ready}; use bytes::Buf; use moq_quic::send::End; @@ -88,18 +88,12 @@ impl Stream { if let Some(err) = shared.closed() { return Poll::Ready(Err(Error::Connection(err))); } - let written = shared.conn.borrow_mut().send_stream(id).write(buf); - match written { - Ok(written) => { - shared.kick(); - Poll::Ready(Ok(written)) - } - // The driver wakes us when moq-quic reports the stream writable, or stopped. - Err(moq_quic::send::Error::Blocked) => { - shared.park_writable(id, waiter); - Poll::Pending - } + // moq-quic wakes us once the stream is writable again, or stopped. + let written = shared.conn.borrow_mut().send_stream(id).poll_write(waiter, buf); + match ready!(written) { + Ok(written) => Poll::Ready(Ok(written)), Err(moq_quic::send::Error::Stopped(code)) => Poll::Ready(Err(Error::Stopped(code))), + Err(moq_quic::send::Error::ConnectionLost) => Poll::Ready(Err(Error::Connection(shared.lost()))), Err(err) => unreachable!("an open stream failed to write: {err:?}"), } } @@ -242,9 +236,7 @@ impl Finished { impl Half { fn finish(&mut self) { // A STOP_SENDING may have beaten us here, which `stopped` reports. - if self.shared.conn.borrow_mut().send_stream(self.id).finish().is_ok() { - self.shared.kick(); - } + let _ = self.shared.conn.borrow_mut().send_stream(self.id).finish(); self.state = HalfState::Finished; } @@ -253,14 +245,10 @@ impl Half { let mut conn = self.shared.conn.borrow_mut(); let mut stream = conn.send_stream(self.id); // Fails harmlessly once the connection is gone. - let reset = match offset { + let _ = match offset { Some(offset) => stream.reset_at(offset, code), None => stream.reset(code), }; - drop(conn); - if reset.is_ok() { - self.shared.kick(); - } } fn set_priority(&mut self, priority: i64) { @@ -289,45 +277,38 @@ impl Half { /// Wait for send stream `id` to end: moq-quic keeps how it ended until its handle releases it. fn poll_stopped(shared: &Shared, id: stream::Id, waiter: &kio::Waiter) -> Poll, Error>> { - let end = shared.conn.borrow_mut().send_stream(id).end(); - match end { - Ok(Some(End::Stopped(code))) => Poll::Ready(Ok(Some(code))), - Ok(Some(_)) => Poll::Ready(Ok(None)), - Ok(None) | Err(_) => match shared.closed() { - Some(err) => Poll::Ready(Err(Error::Connection(err))), - None => { - shared.park_finishing(id, waiter); - Poll::Pending - } - }, + let end = shared.conn.borrow_mut().send_stream(id).poll_end(waiter); + match ready!(end) { + Ok(End::Stopped(code)) => Poll::Ready(Ok(Some(code))), + Ok(_) => Poll::Ready(Ok(None)), + Err(moq_quic::send::Error::ConnectionLost) => Poll::Ready(Err(Error::Connection(shared.lost()))), + // Only a released ID has no end, and the handle releases it only on drop. + Err(err) => unreachable!("a held send stream has no end: {err:?}"), } } impl Drop for Half { fn drop(&mut self) { - self.shared.forget_send(self.id); let mut conn = self.shared.conn.borrow_mut(); let mut stream = conn.send_stream(self.id); let stopped = match stream.end() { Ok(Some(End::Stopped(code))) => Some(code), _ => None, }; - let reset = match (&self.state, stopped) { - (HalfState::Reset, _) => None, + // A reset fails harmlessly once the connection is gone. + let _ = match (&self.state, stopped) { + (HalfState::Reset, _) => Ok(()), // Echo the code, as RFC 9000 section 3.5 asks, even after `finish`: the peer reads // nothing more, so retransmitting the rest is wasted. - (_, Some(code)) => Some(stream.reset(code)), - (HalfState::Finished, None) => None, - (HalfState::Open(offset, code), None) => Some(match offset { + (_, Some(code)) => stream.reset(code), + (HalfState::Finished, None) => Ok(()), + (HalfState::Open(offset, code), None) => match offset { Some(offset) => stream.reset_at(*offset, *code), None => stream.reset(*code), - }), + }, }; stream.release(); drop(conn); - if reset.is_some_and(|reset| reset.is_ok()) { - self.shared.kick(); - } self.shared.release(); } } diff --git a/rs/qmux/Cargo.toml b/rs/qmux/Cargo.toml index c407ca7cbdb..a963848c868 100644 --- a/rs/qmux/Cargo.toml +++ b/rs/qmux/Cargo.toml @@ -27,7 +27,6 @@ kio = { workspace = true, features = ["runtime"] } moq-net = { workspace = true } # The stream state machine, shared with QUIC. `mux` drives it over a reliable transport. moq-quic = { workspace = true, features = ["mux"] } -rustc-hash = { workspace = true } rustls = { version = "0.23", default-features = false, optional = true } thiserror = { workspace = true } tokio = { workspace = true } diff --git a/rs/qmux/src/driver.rs b/rs/qmux/src/driver.rs index 80cca80639f..b647c59caee 100644 --- a/rs/qmux/src/driver.rs +++ b/rs/qmux/src/driver.rs @@ -257,8 +257,6 @@ impl Run { } let now = kio::clock::now(); let mut state = self.inner.lock(); - // Register first: a handle mutating the connection after this re-polls us. - waiter.register(&mut state.driver); for record in self.inbound.drain(..) { if let Ok(record) = &record { self.received_bytes += record.len() as u64; @@ -266,25 +264,30 @@ impl Run { deliver(&mut state, record, now); // Publish what this record did before the next one: a close followed by the // transport ending must report the close. - state.sweep(); + state.check_lost(); } if state.conn.poll_timeout().is_some_and(|at| now >= at) { state.conn.handle_timeout(now); } // Take a batch from the connection only while the transport has room for it, so a - // stalled transport leaves unreliable datagrams in the connection to be shed. + // stalled transport leaves unreliable datagrams in the connection to be shed. With + // nothing to send, `poll_transmit` parks us until a handle queues something. let mut gathered = 0; if self.room && self.outbound.is_empty() { while gathered < BATCH_BYTES { - let Some(record) = state.conn.poll_transmit(now) else { + let Poll::Ready(record) = state.conn.poll_transmit(waiter, now) else { break; }; gathered += record.len(); self.outbound.push_back(record); } } - state.sweep(); + // A close, local or not, starts the close bound, even while the transport holds this + // batch or an earlier one back. `poll_transmit` would take records the transport has no + // room for, so wait for the close with `poll_closed`; the transport wakes us for room. + let _ = state.conn.poll_closed(waiter); + state.check_lost(); if state.conn.is_closed() && self.timer.closing.is_none() { self.timer.closing = Some(now + CLOSE_TIMEOUT); } @@ -400,6 +403,7 @@ fn deliver(state: &mut State, record: Result, now: Instant) { /// The transport is done: publish a local close, or the failure that ended it. fn finish(inner: &Inner, result: Result<(), Error>) { let mut state = inner.lock(); + state.check_lost(); if let Some(err) = state.local_close.take() { state.fail(err); } @@ -407,7 +411,7 @@ fn finish(inner: &Inner, result: Result<(), Error>) { state.fail(err); } state.conn.transport_closed(); - state.sweep(); + state.check_lost(); // A drained connection ended with a close from either side, already published; this only // lands when the transport died under a live one. state.fail(Error::Closed); diff --git a/rs/qmux/src/session.rs b/rs/qmux/src/session.rs index dfde62ae656..c7896ebf18a 100644 --- a/rs/qmux/src/session.rs +++ b/rs/qmux/src/session.rs @@ -2,8 +2,9 @@ //! //! [`mux::Connection`] owns the streams, flow control, and timers, and the //! [`Driver`](crate::Driver) moves its records over the transport. The handles call straight -//! into the connection under its lock and park on the per-stream waiter lists the driver -//! wakes, the way the QUIC drivers do. +//! into the connection under its lock and park on the connection's own waiter lists, which it +//! wakes as the driver feeds it. Whatever a handle queues wakes the driver through +//! [`mux::Connection::poll_transmit`] itself. use std::sync::{Arc, Mutex, MutexGuard}; use std::task::{Poll, ready}; @@ -13,7 +14,6 @@ use bytes::{Buf, Bytes}; use moq_net::transport::{self as net, Address, poll}; use moq_quic::stream::{Dir, Id as StreamId}; use moq_quic::{Side, VarInt, connection::Error as ConnectionError, mux, recv, send}; -use rustc_hash::FxHashMap; use crate::transport::Transport; use crate::{Config, Driver, Error, SharedSocketStats}; @@ -29,115 +29,67 @@ pub(crate) struct Inner { pub(crate) struct State { pub(crate) conn: mux::Connection, - /// Wakes the driver; handles kick it after mutating the connection so fresh egress - /// reaches the transport. - pub(crate) driver: kio::WaiterList, - - established: bool, - establish_waiters: kio::WaiterList, - accept_bi_waiters: kio::WaiterList, - accept_uni_waiters: kio::WaiterList, - /// Whoever is blocked on the peer's MAX_STREAMS credit. - open_waiters: kio::WaiterList, - - /// Per-stream read/write parking, keyed by stream id. - readable: FxHashMap, - writable: FxHashMap, - /// Send streams waiting for their end: a FIN serialized, or a `STOP_SENDING`. - finishing: FxHashMap, - - datagram_waiters: kio::WaiterList, - - /// The terminal error, set exactly once; everything fails with it after. + + /// The terminal error, set exactly once. Handles report it in place of the connection's + /// own reason once it is set, since it can say more (the transport's I/O error, say). + /// + /// The connection's `poll_closed` resolves the moment it is closed, but a close this side + /// asked for is published here only once the driver delivered it and shut the transport, or + /// gave up after [`CLOSE_TIMEOUT`](crate::driver::CLOSE_TIMEOUT) (see `local_close`), so + /// [`Session`]'s `poll_closed` keeps this latch and its own list. + /// Only the driver knows when the close frame reached the transport, which the sans-I/O + /// connection cannot see, and a caller may exit the moment `poll_closed` resolves, so + /// publishing earlier would leave the frame unwritten. closed: Option, closed_waiters: kio::WaiterList, /// The error for a close this side asked for, published once the driver has delivered - /// the close frame, so a caller that exits the moment `poll_closed` resolves already - /// handed it to the transport. + /// the close frame or gave up on a stalled one, so a caller that exits the moment + /// `poll_closed` resolves already handed it to the transport. pub(crate) local_close: Option, } impl State { - /// Terminate with `err` (the first one wins) and wake absolutely everyone. + /// Terminate with `err` (the first one wins), waking whoever waits on the session's + /// `poll_closed`. + /// + /// Every caller loses the connection under the same lock, which wakes everyone parked on + /// it, the driver included. pub(crate) fn fail(&mut self, err: Error) { if self.closed.is_some() { return; } self.closed = Some(err); - self.driver.wake(); self.closed_waiters.wake(); - self.establish_waiters.wake(); - self.accept_bi_waiters.wake(); - self.accept_uni_waiters.wake(); - self.open_waiters.wake(); - self.datagram_waiters.wake(); - for waiters in self.readable.values_mut() { - waiters.wake(); - } - for waiters in self.writable.values_mut() { - waiters.wake(); - } - for waiters in self.finishing.values_mut() { - waiters.wake(); - } - } - - /// Apply every pending connection event to the parking tables. - pub(crate) fn sweep(&mut self) { - while let Some(event) = self.conn.poll() { - match event { - mux::Event::Connected => { - self.established = true; - self.establish_waiters.wake(); - } - mux::Event::Stream(event) => self.sweep_stream(event), - mux::Event::DatagramReceived => self.datagram_waiters.wake(), - mux::Event::ConnectionLost { reason } => { - let local = matches!(reason, ConnectionError::LocallyClosed { .. }); - let err = Error::lost(reason); - match local { - // The driver publishes a local close once its frame is out. - true => self.local_close = Some(err), - false => self.fail(err), - } - } - } - } } - fn sweep_stream(&mut self, event: moq_quic::StreamEvent) { - use moq_quic::StreamEvent; - - // Waking removes the entry: a still-interested poller re-registers on its next poll, - // so the maps only hold streams somebody is parked on. - match event { - StreamEvent::Opened { dir: Dir::Bi } => self.accept_bi_waiters.wake(), - StreamEvent::Opened { dir: Dir::Uni } => self.accept_uni_waiters.wake(), - StreamEvent::Available { .. } => self.open_waiters.wake(), - StreamEvent::Readable { id } => wake(&mut self.readable, id), - StreamEvent::Writable { id } => wake(&mut self.writable, id), - StreamEvent::Finished { id } => wake(&mut self.finishing, id), - StreamEvent::Stopped { id, .. } => { - wake(&mut self.finishing, id); - // A writer blocked on capacity has to learn it will never come. - wake(&mut self.writable, id); - } - // Nothing waits on a send stream's acknowledgment yet. - StreamEvent::Acked { .. } => {} + /// Publish a loss the connection recorded since the last check: a remote close or a + /// violation fails the session now, while a close this side asked for waits for the driver + /// to deliver its frame. + /// + /// Called after every call that can lose the connection: a record, a timeout, a close. + pub(crate) fn check_lost(&mut self) { + if self.closed.is_some() || self.local_close.is_some() { + return; + } + let Some(reason) = self.conn.error() else { + return; + }; + let local = matches!(reason, ConnectionError::LocallyClosed { .. }); + let err = Error::lost(reason.clone()); + match local { + // The driver publishes a local close once its frame is out. + true => self.local_close = Some(err), + false => self.fail(err), } } - /// Drop send stream `id`'s bookkeeping, parking and its end included. - fn forget_send(&mut self, id: StreamId) { - self.finishing.remove(&id); - self.writable.remove(&id); - self.conn.send_stream(id).release(); - } -} - -fn wake(lists: &mut FxHashMap, id: StreamId) { - if let Some(mut waiters) = lists.remove(&id) { - waiters.wake(); + /// What a handle reports once the connection is lost: the session's terminal error, or + /// why the connection ended while a local close is still being delivered. + fn lost(&self) -> Error { + if let Some(err) = &self.closed { + return err.clone(); + } + self.conn.error().cloned().map_or(Error::Closed, Error::lost) } } @@ -152,11 +104,13 @@ impl Inner { if state.closed.is_some() || state.conn.is_closed() { return; } + // Closing wakes everyone parked on the connection. The driver hears it too, through + // `poll_transmit` to deliver the close, or `poll_closed` to start a stalled one's close + // bound. state .conn .close(VarInt::from_u64(code).unwrap_or(VarInt::MAX), reason.to_string()); - state.sweep(); - state.driver.wake(); + state.check_lost(); } } @@ -217,16 +171,6 @@ impl Session { let inner = Arc::new(Inner { state: Mutex::new(State { conn, - driver: kio::WaiterList::new(), - established: false, - establish_waiters: kio::WaiterList::new(), - accept_bi_waiters: kio::WaiterList::new(), - accept_uni_waiters: kio::WaiterList::new(), - open_waiters: kio::WaiterList::new(), - readable: FxHashMap::default(), - writable: FxHashMap::default(), - finishing: FxHashMap::default(), - datagram_waiters: kio::WaiterList::new(), closed: None, closed_waiters: kio::WaiterList::new(), local_close: None, @@ -254,14 +198,7 @@ impl Session { // A finished driver already published why, so only the state needs checking. let _ = waiter.poll_future(std::pin::Pin::new(&mut *driver)); let mut state = inner.lock(); - if state.established { - return Poll::Ready(Ok(())); - } - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - waiter.register(&mut state.establish_waiters); - Poll::Pending + state.conn.poll_connected(waiter).map_err(|_| state.lost()) }) .await?; if self.protocol.is_none() { @@ -294,9 +231,8 @@ impl Session { if capped(limits.receive_rate) || capped(limits.send_rate) { return Err(Error::InvalidConfig("qmux does not pace, so it takes no rate cap")); } - let mut state = self.inner.lock(); - state.conn.set_limits(limits)?; - state.driver.wake(); + // moq-quic wakes the driver for any MAX_DATA or MAX_STREAMS this owes the peer. + self.inner.lock().conn.set_limits(limits)?; Ok(()) } @@ -313,17 +249,7 @@ impl Session { fn poll_accept_dir(&self, waiter: &kio::Waiter, dir: Dir) -> Poll> { let mut state = self.inner.lock(); - if let Some(id) = state.conn.streams().accept(dir) { - return Poll::Ready(Ok(id)); - } - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - match dir { - Dir::Bi => waiter.register(&mut state.accept_bi_waiters), - Dir::Uni => waiter.register(&mut state.accept_uni_waiters), - } - Poll::Pending + state.conn.streams().poll_accept(waiter, dir).map_err(|_| state.lost()) } fn poll_open_dir( @@ -333,17 +259,9 @@ impl Session { options: poll::SendOptions<'_, SendGroup>, ) -> Poll> { let mut state = self.inner.lock(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - // The peer's MAX_STREAMS credit is what gates us: `open` hands back nothing while it - // is spent. - if let Some(id) = state.conn.streams().open(dir, to_quic(options)) { - state.driver.wake(); - return Poll::Ready(Ok(id)); - } - waiter.register(&mut state.open_waiters); - Poll::Pending + // The peer's MAX_STREAMS credit is what gates us: this parks while it is spent. + let id = ready!(state.conn.streams().poll_open(waiter, dir, to_quic(options))).map_err(|_| state.lost())?; + Poll::Ready(Ok(id)) } } @@ -410,19 +328,13 @@ impl poll::Session for Session { options: poll::SendOptions<'_, SendGroup>, ) -> Poll>, Error>> { let mut state = self.inner.lock(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); + if state.conn.is_closed() { + return Poll::Ready(Err(state.lost())); } // A session started on an out-of-band protocol (a WebSocket subprotocol) runs before // the peer's transport parameters arrive. Until then its credit is unknown, not spent. - if !state.established { - waiter.register(&mut state.establish_waiters); - return Poll::Pending; - } + ready!(state.conn.poll_connected(waiter)).map_err(|_| state.lost())?; let id = state.conn.streams().open(Dir::Bi, to_quic(options)); - if id.is_some() { - state.driver.wake(); - } drop(state); Poll::Ready(Ok(id.map(|id| { ( @@ -446,20 +358,12 @@ impl poll::Session for Session { Err(send::Error::TooLarge) => return Err(Error::DatagramTooLarge), Err(_) => return Err(Error::DatagramsUnsupported), } - state.driver.wake(); Ok(()) } fn poll_recv_datagram(&self, waiter: &kio::Waiter) -> Poll> { let mut state = self.inner.lock(); - if let Some(datagram) = state.conn.datagrams().recv() { - return Poll::Ready(Ok(datagram)); - } - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - waiter.register(&mut state.datagram_waiters); - Poll::Pending + state.conn.datagrams().poll_recv(waiter).map_err(|_| state.lost()) } fn max_datagram_size(&self) -> usize { @@ -558,7 +462,6 @@ impl poll::SendGroup for SendGroup { fn set_priority(&mut self, priority: i64) { let mut state = self.0.inner.lock(); state.conn.send_groups().set_priority(self.id(), priority); - state.driver.wake(); } } @@ -605,7 +508,6 @@ impl SendStream { let mut state = self.inner.lock(); // Err means the stream is already gone, which is what we wanted. let _ = state.conn.send_stream(self.id).reset(code); - state.driver.wake(); self.reset = true; } } @@ -623,20 +525,11 @@ impl poll::SendStream for SendStream { return Poll::Ready(Err(Error::StreamClosed)); } let mut state = self.inner.lock(); - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } // Takes only what the connection accepts, which for a `Bytes` source is no copy at all. - match state.conn.send_stream(self.id).write(buf) { - Ok(n) => { - state.driver.wake(); - Poll::Ready(Ok(n)) - } - Err(send::Error::Blocked) => { - waiter.register(state.writable.entry(self.id).or_default()); - Poll::Pending - } + match ready!(state.conn.send_stream(self.id).poll_write(waiter, buf)) { + Ok(n) => Poll::Ready(Ok(n)), Err(send::Error::Stopped(code)) => Poll::Ready(Err(Error::StreamStop(code.into_inner()))), + Err(send::Error::ConnectionLost) => Poll::Ready(Err(state.lost())), Err(_) => Poll::Ready(Err(Error::StreamClosed)), } } @@ -659,7 +552,7 @@ impl poll::SendStream for SendStream { match state.conn.send_stream(self.id).finish() { Ok(()) => {} // A STOP_SENDING beat us here; carry its code like `poll_write` does. The stream is - // over without our reset, so `poll_closed` reports the stop the sweep recorded. + // over without our reset, so `poll_closed` reports the stop the connection recorded. Err(send::Error::Stopped(code)) => { self.fin = true; return Err(Error::StreamStop(code.into_inner())); @@ -667,7 +560,6 @@ impl poll::SendStream for SendStream { // Already finished or reset, so the FIN it wanted is out. Err(_) => {} } - state.driver.wake(); self.fin = true; Ok(()) } @@ -683,18 +575,14 @@ impl poll::SendStream for SendStream { let mut state = self.inner.lock(); // A finish is acknowledged once its FIN goes to the transport, which on a reliable // transport is delivery. - match state.conn.send_stream(self.id).end() { - Ok(Some(send::End::Stopped(code))) => Poll::Ready(Err(Error::StreamStop(code.into_inner()))), - Ok(Some(_)) => Poll::Ready(Ok(())), + match ready!(state.conn.send_stream(self.id).poll_end(waiter)) { + Ok(send::End::Stopped(code)) => Poll::Ready(Err(Error::StreamStop(code.into_inner()))), + Ok(_) => Poll::Ready(Ok(())), // The end never came. If the session died first, the caller cannot read success // as "every byte arrived". - Ok(None) | Err(_) => match &state.closed { - Some(err) => Poll::Ready(Err(err.clone())), - None => { - waiter.register(state.finishing.entry(self.id).or_default()); - Poll::Pending - } - }, + Err(send::Error::ConnectionLost) => Poll::Ready(Err(state.lost())), + // The ID is only released on drop, so this does not happen. + Err(_) => Poll::Ready(Err(Error::StreamClosed)), } } } @@ -705,9 +593,8 @@ impl Drop for SendStream { // Reset before giving up the ID, which refuses every later call on it. if !self.fin && !self.reset { let _ = state.conn.send_stream(self.id).reset(VarInt::from_u32(0)); - state.driver.wake(); } - state.forget_send(self.id); + state.conn.send_stream(self.id).release(); } } @@ -753,33 +640,24 @@ impl RecvStream { return Poll::Ready(Err(Error::StreamClosed)); } let mut state = self.inner.lock(); - let (result, transmit) = state.conn.recv_stream(self.id).lease(want); - // Freed credit, or an eviction whose reader the driver's sweep wakes. - if transmit.should_transmit() { - state.driver.wake(); - } - match result { + // moq-quic wakes the driver for any credit the lease frees, and the reader of any stream + // it evicts. + let (result, _) = state.conn.recv_stream(self.id).poll_lease(waiter, want); + match ready!(result) { Ok(lease) => Poll::Ready(Ok(lease)), - Err(recv::Error::Blocked) => { - if let Some(err) = &state.closed { - return Poll::Ready(Err(err.clone())); - } - waiter.register(state.readable.entry(self.id).or_default()); - Poll::Pending - } Err(recv::Error::Reset(code)) => { // Observing the reset frees the stream and the credit its lease withheld; only // then may later calls skip it. let _ = state.conn.recv_stream(self.id).received_reset(); - state.readable.remove(&self.id); - state.driver.wake(); self.reset = Some(code.into_inner()); Poll::Ready(Err(Error::StreamReset(code.into_inner()))) } Err(recv::Error::Evicted) => Poll::Ready(Err(Error::Evicted)), Err(recv::Error::TooLarge) => Poll::Ready(Err(Error::LeaseTooLarge)), - // The stream is gone, which only a lost connection does to one we hold. - Err(_) => Poll::Ready(Err(state.closed.clone().unwrap_or(Error::Closed))), + // The connection was lost, once everything that arrived before it was taken. The + // stream being gone means the same, since only a lost connection does that to one + // we hold. + Err(_) => Poll::Ready(Err(state.lost())), } } } @@ -808,10 +686,8 @@ impl poll::RecvStream for RecvStream { return; } let mut state = self.inner.lock(); - state.readable.remove(&self.id); // Err means the stream is already gone, which is what we wanted. let _ = state.conn.recv_stream(self.id).stop(VarInt::from_u32(code)); - state.driver.wake(); self.stopped = true; } @@ -823,25 +699,18 @@ impl poll::RecvStream for RecvStream { return Poll::Ready(Error::StreamClosed); } let mut state = self.inner.lock(); - match state.conn.recv_stream(self.id).received_reset() { + match ready!(state.conn.recv_stream(self.id).poll_received_reset(waiter)) { + Ok(code) => { + // Observing it freed the stream, and moq-quic wakes the driver for any credit + // that owes the peer. + self.reset = Some(code.into_inner()); + Poll::Ready(Error::StreamReset(code.into_inner())) + } // Evicted, so its bytes are gone: nothing frees the window until it is stopped. Err(recv::Error::Evicted) => Poll::Ready(Error::Evicted), + Err(recv::Error::ConnectionLost) => Poll::Ready(state.lost()), // Every byte arrived, so nothing is left to reset. Err(_) => Poll::Pending, - Ok(Some(code)) => { - // Observing it freed the stream, which may owe the peer credit. - self.reset = Some(code.into_inner()); - state.driver.wake(); - Poll::Ready(Error::StreamReset(code.into_inner())) - } - Ok(None) => { - if let Some(err) = &state.closed { - return Poll::Ready(err.clone()); - } - // A reset notifies readers, since it is an immediate read error. - waiter.register(state.readable.entry(self.id).or_default()); - Poll::Pending - } } } } @@ -852,11 +721,9 @@ impl Drop for RecvStream { return; } let mut state = self.inner.lock(); - state.readable.remove(&self.id); // A leased stream keeps its state until stopped, even once it ended; Err means it is // already gone. let _ = state.conn.recv_stream(self.id).stop(VarInt::from_u32(0)); - state.driver.wake(); } } @@ -917,9 +784,8 @@ impl poll::Lease for RecvLease<'_> { fn release(mut self) { let lease = self.lease.take().expect("only taken to hand it back"); let mut state = self.stream.inner.lock(); - if state.conn.recv_stream(self.stream.id).release(lease).should_transmit() { - state.driver.wake(); - } + // moq-quic wakes the driver for the credit this returns. + let _ = state.conn.recv_stream(self.stream.id).release(lease); } } @@ -929,9 +795,8 @@ impl Drop for RecvLease<'_> { return; }; let mut state = self.stream.inner.lock(); - if state.conn.recv_stream(self.stream.id).restore(lease).should_transmit() { - state.driver.wake(); - } + // moq-quic wakes the driver if the window was stuck on this lease. + let _ = state.conn.recv_stream(self.stream.id).restore(lease); } } diff --git a/rs/qmux/src/session/tests.rs b/rs/qmux/src/session/tests.rs index 1aca3608374..fda53695a21 100644 --- a/rs/qmux/src/session/tests.rs +++ b/rs/qmux/src/session/tests.rs @@ -596,6 +596,25 @@ async fn a_close_ends_an_unfinished_read_after_its_data() { drop(send); } +/// Waits parked on the closing side end with its close instead of hanging. +#[tokio::test(start_paused = true)] +async fn a_local_close_ends_parked_waits() { + let (_client, server) = pair(config(Version::QMux02), config(Version::QMux02)).await; + let (_send, mut recv) = server.open_bi(SendOptions::default()).await.unwrap(); + let mut accepting = std::pin::pin!(server.accept_uni()); + let mut reading = std::pin::pin!(recv.read_chunk(usize::MAX)); + assert!(futures_poll_once(&mut accepting).await.is_none()); + assert!(futures_poll_once(&mut reading).await.is_none()); + + server.close(42, "bye"); + assert!(matches!(accepting.await, Err(Error::ConnectionClosed { code: 42, .. }))); + let read = reading.await; + assert!( + matches!(read, Err(Error::ConnectionClosed { code: 42, .. })), + "got {read:?}" + ); +} + #[tokio::test(start_paused = true)] async fn dropping_the_last_handle_closes_the_session() { let (client, server) = pair(config(Version::QMux01), config(Version::QMux01)).await; diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index 56bcd8ff81f..ec45768f827 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -124,10 +124,10 @@ case "$1" in # The allocation/window/play-buffer/receive-bursts tables keep their stdout in the run; # only Criterion estimates enter the trend report. criterion extra --no-fail-fast -p moq-e2ee -p moq-sign -p moq-json -p moq-archive -p moq-cli -p moq-mux -p moq-audio \ - -p moq-quic -p moq-sock -p moq-transcode --features moq-quic/fuzz \ + -p moq-quic -p moq-sock -p moq-tokio -p moq-transcode --features moq-quic/fuzz \ --bench protect --bench sign --bench codec --bench table --bench allocations --bench window \ --bench producer --bench play_buffer --bench jitter --bench decode_clock --bench ts_programs --bench ts_video \ - --bench bus --bench parameter_sets --bench sent_packets --bench demux --bench receive_bursts --bench ladder + --bench bus --bench parameter_sets --bench sent_packets --bench demux --bench receive_bursts --bench ladder --bench fanout # moq-auth's published release is its own dev-dependency, so `-p moq-auth` is ambiguous. # The concurrent cases' latency percentiles stay in the run's stdout. criterion auth --manifest-path rs/moq-auth/Cargo.toml --bench verify