diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index f25d1eefb69..f91ea550921 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -31,8 +31,8 @@ This README's own work, after the children: run `just check --all` several times on a loaded machine, as the first round did. Done 2026-11-13 on fedora (32 cores, load average 4 to 14 from other agents' sessions): three runs all passed. The one failure seen on that host the same day, moq-relay's -io_uring `ENOMEM` at ring setup, is planned in -[uring-setup-enomem](/quest/m0/test-flakes-2/uring-setup-enomem.md). +io_uring `ENOMEM` at ring setup, was fixed by shrinking the CQ so the rings +region stays within order 3. Decided 2026-11-13 (maintainer, `/quest-complete`): that child stays in m0 under this epic, so the epic stays open until it lands (rejected: moving it to m1, moving it to m3). The epic completes when that child lands. diff --git a/quest/m0/test-flakes-2/uring-setup-enomem.md b/quest/m0/test-flakes-2/uring-setup-enomem.md deleted file mode 100644 index 3acf33db539..00000000000 --- a/quest/m0/test-flakes-2/uring-setup-enomem.md +++ /dev/null @@ -1,42 +0,0 @@ -# [S] io_uring ring setup survives a fragmented host - -## Goal - -moq-relay's io_uring tests, such as `bitrate_caps::uring_session_is_held_to_its_caps`, -stop failing at `moq_uring::Worker::new` with `ENOMEM` on a loaded Linux 7.3 -host, and the relay's startup error names the cause the kernel actually hit. - -## Plan - -Seen 2026-11-13 on fedora (Linux 7.3.3, `ulimit -l` 8192) during a loaded -`just check`: the test's capability probe got `Cannot allocate memory (os error -12)` from `io_uring_setup` within 10 ms, and the rerun passed. - -It was not `RLIMIT_MEMLOCK`, despite the message blaming it. Since -[moq-dev/moq#8641](https://github.com/moq-dev/moq/pull/8641) showed 7.3.3 sets -up a ring under a limit of 0, and v7.3's `io_uring.c` creates both ring regions -with `io_create_region(NULL, ...)`, so nothing is charged to the user. The -host's memory cgroups recorded no `max` or `oom` events either. - -The likely cause, from v7.3's `io_uring/memmap.c` `io_region_allocate_pages`: -it first tries one compound allocation of `get_order(size)`, and on failure -falls back to `alloc_pages_bulk_node`, returning `ENOMEM` whenever the bulk -call hands back fewer pages than asked. The bulk allocator returns a partial -batch rather than reclaiming. The production CQ of 2048 entries is 32 KiB of -CQEs plus the ring header and SQ array, so the region spills just past 32 KiB -into an order-4 (64 KiB) compound allocation, above `PAGE_ALLOC_COSTLY_ORDER`, -which the page allocator gives up on quickly when memory is fragmented. - -Open (implementer): confirm the cause, for example by fragmenting memory or -tracing `io_region_allocate_pages`'s returns while the test loops, before -changing code. Candidate fixes, in order of preference: - -- Size the rings so each region stays at or below order 3 (32 KiB). The - rings region holds the header and SQ array besides the CQEs, and 2048 CQEs - already fill 32 KiB, so `IORING_SETUP_NO_SQARRAY` alone cannot do it: the CQ - must shrink too, such as to 1024 entries. This touches the CQ sizing - rationale in `rs/moq-uring/src/worker.rs`. -- Report it upstream if the bulk fallback's partial return is the whole cause. - -Never retry setup in the test. `Error::ring`'s message should stop naming -`RLIMIT_MEMLOCK` first on kernels that do not charge it. diff --git a/rs/moq-uring/README.md b/rs/moq-uring/README.md index 16220125551..90f2e9272d3 100644 --- a/rs/moq-uring/README.md +++ b/rs/moq-uring/README.md @@ -156,7 +156,7 @@ Linux 6.14 up to the 7.3 fix (backported to stable kernels) charges rings to locked memory per user across processes, not per nextest run; other kernels charge them to the memory cgroup. Unit tests use 128 SQ entries and 512 CQ entries instead of the production -256/2048 geometry. Integration tests still exercise production rings. +256/1024 geometry. Integration tests still exercise production rings. The nextest group allows four ring tests per invocation. Each socket-queue benchmark axis occupies all four slots, excluding other ring tests while measuring. @@ -164,7 +164,7 @@ Other programs draw on the same budget. libuv gives every event loop a 256-entry ring of 28 KiB, and it creates these without checking `UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an 8 MiB limit before any test started, mostly in Node.js tool servers. A -production-geometry worker takes 52 KiB and a unit-test worker 20 KiB. +production-geometry worker takes 36 KiB and a unit-test worker 20 KiB. One `moq-uring` plus `moq-relay` test invocation needs about 0.5 to 1 MiB of what is left. Count the rings with `ls -l /proc/*/fd 2>/dev/null | grep -c io_uring`. diff --git a/rs/moq-uring/src/error.rs b/rs/moq-uring/src/error.rs index 45922ac0beb..0aa798fa688 100644 --- a/rs/moq-uring/src/error.rs +++ b/rs/moq-uring/src/error.rs @@ -15,13 +15,12 @@ pub enum Error { } impl Error { - /// A failed ring setup or registration, naming the locked-memory limit when - /// that is what ran out. + /// A failed ring setup or registration, naming what an `ENOMEM` can mean. /// - /// Linux 6.14 up to the 7.3 fix (backported to stable) charges every ring - /// and provided-buffer ring to the user's `RLIMIT_MEMLOCK`, shared with - /// every other io_uring that user runs, so a bare `ENOMEM` points at the - /// wrong culprit. Other kernels charge them to the memory cgroup. + /// The kernel cannot tell which ran out, and a stable backport makes the + /// version no guide: the memory cgroup, physically contiguous pages on a + /// fragmented host, or, from Linux 6.14 until the 7.3 fix, the user's + /// `RLIMIT_MEMLOCK`, shared with every other io_uring that user runs. pub(crate) fn ring(err: std::io::Error) -> Self { if err.raw_os_error() != Some(libc::ENOMEM) { return Self::Io(err); @@ -38,9 +37,9 @@ impl Error { Self::Io(std::io::Error::new( err.kind(), format!( - "{err}: Linux 6.14 until the 7.3 fix charges io_uring rings to RLIMIT_MEMLOCK ({limit}), shared by every \ - process of this user, so raise it with `ulimit -l` or systemd's `LimitMEMLOCK=`; other kernels charge \ - them to the memory cgroup" + "{err}: the kernel could not allocate the io_uring ring, from the memory cgroup or from contiguous pages \ + on a fragmented host; Linux 6.14 until the 7.3 fix also charges it to RLIMIT_MEMLOCK ({limit}), shared \ + by every process of this user, which `ulimit -l` or systemd's `LimitMEMLOCK=` raises" ), )) } @@ -51,9 +50,12 @@ mod tests { use super::*; #[test] - fn a_ring_enomem_names_the_memlock_limit() { + fn a_ring_enomem_names_each_cause() { let err = anyhow::Error::new(Error::ring(std::io::Error::from_raw_os_error(libc::ENOMEM))); - assert!(format!("{err:#}").contains("RLIMIT_MEMLOCK"), "{err:#}"); + let msg = format!("{err:#}"); + for cause in ["memory cgroup", "fragmented", "RLIMIT_MEMLOCK"] { + assert!(msg.contains(cause), "{msg}"); + } let err = anyhow::Error::new(Error::ring(std::io::Error::from_raw_os_error(libc::EBADF))); assert!(!format!("{err:#}").contains("RLIMIT_MEMLOCK"), "{err:#}"); diff --git a/rs/moq-uring/src/udp.rs b/rs/moq-uring/src/udp.rs index e13c3b5dab9..dea86ff9835 100644 --- a/rs/moq-uring/src/udp.rs +++ b/rs/moq-uring/src/udp.rs @@ -140,11 +140,12 @@ impl Default for Config { gro: true, gso: true, multishot: true, - // 16 MiB and 64 MiB of headroom at the default buffer lengths, - // reached only by a socket that actually starves for them. + // 16 MiB and 32 MiB of headroom at the default buffer lengths, + // reached only by a socket that actually starves for them. Their + // sum stays under the worker's CQ, which can grow no further. rx_buffers_max: 256, rx_buffer_len: MAX_RECV + RECV_OVERHEAD, - tx_buffers_max: 1024, + tx_buffers_max: 512, tx_buffer_len: 64 * 1024, } } diff --git a/rs/moq-uring/src/worker.rs b/rs/moq-uring/src/worker.rs index 4cf167c37b1..8f3718ace75 100644 --- a/rs/moq-uring/src/worker.rs +++ b/rs/moq-uring/src/worker.rs @@ -20,8 +20,9 @@ use crate::{Error, udp}; /// Submission queue depth. The SQ only holds SQEs staged between submits, /// never in-flight operations, so it needs no relation to the socket pools; /// [`Shared::push`] submits inline whenever it fills. +const PRODUCTION_SQ_ENTRIES: u32 = 256; #[cfg(not(test))] -const SQ_ENTRIES: u32 = 256; +const SQ_ENTRIES: u32 = PRODUCTION_SQ_ENTRIES; #[cfg(test)] const SQ_ENTRIES: u32 = 128; @@ -34,10 +35,13 @@ const SQ_ENTRIES: u32 = 128; /// allocation-per-CQE slow path and it ends any armed multishot receive, so /// the CQ is sized to keep it out of steady state. /// -/// No larger: the ring costs 16 bytes per entry, most of each worker's -/// footprint, and Linux 6.14 up to the 7.3 fix charges it to `RLIMIT_MEMLOCK`, -/// a budget shared by every io_uring the user runs. -const PRODUCTION_CQ_ENTRIES: u32 = 2048; +/// No larger: the kernel allocates the CQEs, the ring header, and the SQ +/// array as one region, first as one compound allocation and then as single +/// pages that the bulk allocator may hand back short, failing setup with +/// `ENOMEM`. At 1024 entries the region fits order 3 (32 KiB), which the page +/// allocator reclaims for rather than give up on; 2048 spills into order 4, +/// which fails on a fragmented host. +const PRODUCTION_CQ_ENTRIES: u32 = 1024; #[cfg(not(test))] const CQ_ENTRIES: u32 = PRODUCTION_CQ_ENTRIES; // Unit tests exercise overflow handling but do not need production pool capacity. @@ -1493,6 +1497,17 @@ mod tests { ); } + #[test] + fn the_rings_region_stays_within_order_3() { + // v7.3 `rings_size`: a cacheline header, the CQEs, then the SQ array + // at the next cacheline. Past `PAGE_ALLOC_COSTLY_ORDER` the compound + // allocation gives up under fragmentation and setup can fail. + const CACHELINE: u32 = 64; + let cqes = CACHELINE + PRODUCTION_CQ_ENTRIES * 16; + let rings = cqes.next_multiple_of(CACHELINE) + PRODUCTION_SQ_ENTRIES * 4; + assert!(rings <= 32 * 1024, "the rings region needs {rings} bytes, past order 3"); + } + #[test] fn the_ring_honors_the_requested_cq_depth() { // The kernel-reported geometry, not the constant: dropping the