Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions quest/m0/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ This README's own work, after the children: run `just check --all` several
times on a loaded machine, as the first round did. Done 2026-11-13 on fedora
(32 cores, load average 4 to 14 from other agents' sessions): three runs all
passed. The one failure seen on that host the same day, moq-relay's
io_uring `ENOMEM` at ring setup, is planned in
[uring-setup-enomem](/quest/m0/test-flakes-2/uring-setup-enomem.md).
io_uring `ENOMEM` at ring setup, was fixed by shrinking the CQ so the rings
region stays within order 3.
Decided 2026-11-13 (maintainer, `/quest-complete`): that child stays in m0
under this epic, so the epic stays open until it lands (rejected: moving it to
m1, moving it to m3). The epic completes when that child lands.
Expand Down
42 changes: 0 additions & 42 deletions quest/m0/test-flakes-2/uring-setup-enomem.md

This file was deleted.

4 changes: 2 additions & 2 deletions rs/moq-uring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,15 +156,15 @@ Linux 6.14 up to the 7.3 fix (backported to stable kernels) charges rings to
locked memory per user across processes, not per nextest run; other kernels
charge them to the memory cgroup.
Unit tests use 128 SQ entries and 512 CQ entries instead of the production
256/2048 geometry. Integration tests still exercise production rings.
256/1024 geometry. Integration tests still exercise production rings.
The nextest group allows four ring tests per invocation. Each socket-queue
benchmark axis occupies all four slots, excluding other ring tests while measuring.

Other programs draw on the same budget. libuv gives every event loop a
256-entry ring of 28 KiB, and it creates these without checking
`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an
8 MiB limit before any test started, mostly in Node.js tool servers. A
production-geometry worker takes 52 KiB and a unit-test worker 20 KiB.
production-geometry worker takes 36 KiB and a unit-test worker 20 KiB.
One `moq-uring` plus `moq-relay` test invocation needs about 0.5 to 1 MiB
of what is left. Count the rings with `ls -l /proc/*/fd 2>/dev/null | grep -c io_uring`.

Expand Down
24 changes: 13 additions & 11 deletions rs/moq-uring/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,12 @@ pub enum Error {
}

impl Error {
/// A failed ring setup or registration, naming the locked-memory limit when
/// that is what ran out.
/// A failed ring setup or registration, naming what an `ENOMEM` can mean.
///
/// Linux 6.14 up to the 7.3 fix (backported to stable) charges every ring
/// and provided-buffer ring to the user's `RLIMIT_MEMLOCK`, shared with
/// every other io_uring that user runs, so a bare `ENOMEM` points at the
/// wrong culprit. Other kernels charge them to the memory cgroup.
/// The kernel cannot tell which ran out, and a stable backport makes the
/// version no guide: the memory cgroup, physically contiguous pages on a
/// fragmented host, or, from Linux 6.14 until the 7.3 fix, the user's
/// `RLIMIT_MEMLOCK`, shared with every other io_uring that user runs.
pub(crate) fn ring(err: std::io::Error) -> Self {
if err.raw_os_error() != Some(libc::ENOMEM) {
return Self::Io(err);
Expand All @@ -38,9 +37,9 @@ impl Error {
Self::Io(std::io::Error::new(
err.kind(),
format!(
"{err}: Linux 6.14 until the 7.3 fix charges io_uring rings to RLIMIT_MEMLOCK ({limit}), shared by every \
process of this user, so raise it with `ulimit -l` or systemd's `LimitMEMLOCK=`; other kernels charge \
them to the memory cgroup"
"{err}: the kernel could not allocate the io_uring ring, from the memory cgroup or from contiguous pages \
on a fragmented host; Linux 6.14 until the 7.3 fix also charges it to RLIMIT_MEMLOCK ({limit}), shared \
by every process of this user, which `ulimit -l` or systemd's `LimitMEMLOCK=` raises"
),
))
}
Expand All @@ -51,9 +50,12 @@ mod tests {
use super::*;

#[test]
fn a_ring_enomem_names_the_memlock_limit() {
fn a_ring_enomem_names_each_cause() {
let err = anyhow::Error::new(Error::ring(std::io::Error::from_raw_os_error(libc::ENOMEM)));
assert!(format!("{err:#}").contains("RLIMIT_MEMLOCK"), "{err:#}");
let msg = format!("{err:#}");
for cause in ["memory cgroup", "fragmented", "RLIMIT_MEMLOCK"] {
assert!(msg.contains(cause), "{msg}");
}

let err = anyhow::Error::new(Error::ring(std::io::Error::from_raw_os_error(libc::EBADF)));
assert!(!format!("{err:#}").contains("RLIMIT_MEMLOCK"), "{err:#}");
Expand Down
7 changes: 4 additions & 3 deletions rs/moq-uring/src/udp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -140,11 +140,12 @@ impl Default for Config {
gro: true,
gso: true,
multishot: true,
// 16 MiB and 64 MiB of headroom at the default buffer lengths,
// reached only by a socket that actually starves for them.
// 16 MiB and 32 MiB of headroom at the default buffer lengths,
// reached only by a socket that actually starves for them. Their
// sum stays under the worker's CQ, which can grow no further.
rx_buffers_max: 256,
rx_buffer_len: MAX_RECV + RECV_OVERHEAD,
tx_buffers_max: 1024,
tx_buffers_max: 512,
tx_buffer_len: 64 * 1024,
}
}
Expand Down
25 changes: 20 additions & 5 deletions rs/moq-uring/src/worker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,9 @@ use crate::{Error, udp};
/// Submission queue depth. The SQ only holds SQEs staged between submits,
/// never in-flight operations, so it needs no relation to the socket pools;
/// [`Shared::push`] submits inline whenever it fills.
const PRODUCTION_SQ_ENTRIES: u32 = 256;
#[cfg(not(test))]
const SQ_ENTRIES: u32 = 256;
const SQ_ENTRIES: u32 = PRODUCTION_SQ_ENTRIES;
#[cfg(test)]
const SQ_ENTRIES: u32 = 128;

Expand All @@ -34,10 +35,13 @@ const SQ_ENTRIES: u32 = 128;
/// allocation-per-CQE slow path and it ends any armed multishot receive, so
/// the CQ is sized to keep it out of steady state.
///
/// No larger: the ring costs 16 bytes per entry, most of each worker's
/// footprint, and Linux 6.14 up to the 7.3 fix charges it to `RLIMIT_MEMLOCK`,
/// a budget shared by every io_uring the user runs.
const PRODUCTION_CQ_ENTRIES: u32 = 2048;
/// No larger: the kernel allocates the CQEs, the ring header, and the SQ
/// array as one region, first as one compound allocation and then as single
/// pages that the bulk allocator may hand back short, failing setup with
/// `ENOMEM`. At 1024 entries the region fits order 3 (32 KiB), which the page
/// allocator reclaims for rather than give up on; 2048 spills into order 4,
/// which fails on a fragmented host.
const PRODUCTION_CQ_ENTRIES: u32 = 1024;
#[cfg(not(test))]
const CQ_ENTRIES: u32 = PRODUCTION_CQ_ENTRIES;
// Unit tests exercise overflow handling but do not need production pool capacity.
Expand Down Expand Up @@ -1493,6 +1497,17 @@ mod tests {
);
}

#[test]
fn the_rings_region_stays_within_order_3() {
// v7.3 `rings_size`: a cacheline header, the CQEs, then the SQ array
// at the next cacheline. Past `PAGE_ALLOC_COSTLY_ORDER` the compound
// allocation gives up under fragmentation and setup can fail.
const CACHELINE: u32 = 64;
let cqes = CACHELINE + PRODUCTION_CQ_ENTRIES * 16;
let rings = cqes.next_multiple_of(CACHELINE) + PRODUCTION_SQ_ENTRIES * 4;
assert!(rings <= 32 * 1024, "the rings region needs {rings} bytes, past order 3");
}

#[test]
fn the_ring_honors_the_requested_cq_depth() {
// The kernel-reported geometry, not the constant: dropping the
Expand Down