Skip to content

fix(moq-uring): keep the rings region within order 3 so setup survives fragmentation - #8718

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-2/uring-setup-enomem
Nov 14, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-2/uring-setup-enomem

Conversation

@kixelated

@kixelated kixelated commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

On a loaded host, moq-relay's io_uring tests sometimes failed at moq_uring::Worker::new because io_uring_setup returned ENOMEM. In v7.3, io_region_allocate_pages first tries one compound allocation of get_order(size). If that fails, it falls back to alloc_pages_bulk_node, and that call can return fewer pages than asked. With 2048 CQ entries the rings region came to 33,856 B, just past 32 KiB, so it needed an order-4 allocation. Order 4 is above PAGE_ALLOC_COSTLY_ORDER, so the allocator gives up on it quickly when memory is fragmented.

  • The production CQ goes from 2048 to 1024 entries, making the region 17,472 B (order 3, which the allocator reclaims for). A new unit test guards that bound.
  • udp::Config::default().tx_buffers_max goes from 1024 to 512. With 256 rx buffers, the default pool ceilings still fit in the CQ, which cq_covers_the_default_pool_ceilings checks.
  • Error::ring's ENOMEM message now lists the memory cgroup and fragmented contiguous pages before RLIMIT_MEMLOCK, which only Linux 6.14 up to the 7.3 fix charges.
  • Removes the finished quest and updates the epic README.

Decisions

All decisions here were taken by the agent under /quest-afk, for the maintainer's review.

  • Fix: CQ 1024 plus tx ceiling 512. Rejected options:
    • IORING_SETUP_NO_SQARRAY alone: it still leaves 32,832 B, which is order 4.
    • Keeping CQ 2048 and dropping the tx ceiling to 768, as an alternative coverage target: the CQ alone is already order 4.
    • IORING_SETUP_NO_MMAP with user-allocated rings: io-uring 0.7.15 has no builder for it.
    • Retrying setup: the quest forbids it.
  • tx 512, not lower rx. That leaves 32 MiB of GSO trains in flight per socket. rx, at 256, already scales with the connection count.
  • Confirmation by source and layout, not induced fragmentation. Fragmenting memory on fedora unprivileged means eating most of the host's RAM next to CI runners and agent sessions, and fault injection needs root. Rejected: running that experiment on the shared host, or building a VM for it, which is out of scope for an S quest.
  • No upstream report filed. The bulk fallback's partial return is a kernel weakness, but it can't be hit at order 3. Rejected: filing one as part of this quest.
  • Pullfrog's README note is fixed. rs/moq-uring/README.md now says 256/1024 and 36 KiB per production worker (17,472 B of rings in 5 pages plus 16 KiB of SQEs in 4), which is consistent with its 20 KiB unit-test figure.

Validation

  • Ran io_uring_setup directly on Linux 7.3.3 (fedora) with these flags: the kernel reports cq_off.cqes = 64. Following v7.3's rings_size, the region is 64 + 2048·16 = 32,832 B before the SQ array, so order 4 even with NO_SQARRAY. With 1024 entries it is 17,472 B with the SQ array, so order 3. This confirms More tests hold up under load #8670's CodeRabbit correction.
  • Read v7.3's io_uring/memmap.c: the compound attempt, then the bulk fallback that returns ENOMEM on a short count.
  • cargo test -p moq-uring --lib: 79 passed.
  • cargo test -p moq-relay --features io-uring --test integration uring_: 17 passed, including bitrate_caps::uring_session_is_held_to_its_caps.
  • just check passed.
  • Not measured: whether halving the tx ceiling costs throughput on a socket that saturates it. No bench here sweeps tx_buffers_max.

Impact

  • Public API: none.
  • Wire: none.
  • Behavior: the default moq_uring::udp::Config::tx_buffers_max drops from 1024 to 512, and each worker's CQ halves.

Caused by

  • moq-dev/moq#3133 - sized the CQ at 4096 (later 2048), past order 3, to cover a tx ceiling of 1024.

Models

Size: S
Implementer: Claude, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 2a16a28

  • Base: origin/main at 6f8b127

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1d85c-bebe-7b81-893c-e1a062ef629c, round 2

  • Score: 9/10, The incremental changes only correct README ring geometry and memory figures to match the implementation, with no actionable findings; builds and tests were not run as instructed.

  • Rounds: 2; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 13, 2026 20:09
…s fragmentation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title io_uring ring setup survives a fragmented host fix(moq-uring): keep the rings region within order 3 so setup survives fragmentation Nov 14, 2026
@kixelated
kixelated marked this pull request as ready for review November 14, 2026 04:42

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one stale doc outside the diff.

Reviewed changes

Reviewed the full diff: shrinking the production io_uring CQ so the rings region stays within order 3, the matching tx pool ceiling, and the reworded ENOMEM message.

  • CQ 2048 → 1024: PRODUCTION_CQ_ENTRIES halves, so the rings region drops from 33,856 B (order 4) to 17,472 B (order 3, the PAGE_ALLOC_COSTLY_ORDER bound the allocator still reclaims for). The arithmetic in the_rings_region_stays_within_order_3 matches v7.3 rings_size with the 64 B header the author measured.
  • tx_buffers_max 1024 → 512: keeps cq_covers_the_default_pool_ceilings true (512 + 256 < 1024).
  • PRODUCTION_SQ_ENTRIES: split out so the new test reads the production SQ depth under cfg(test).
  • Error::ring message: lists the memory cgroup and fragmented contiguous pages before RLIMIT_MEMLOCK, and the test checks all three.
  • Quest bookkeeping: removes the finished quest and updates the epic README.

ℹ️ rs/moq-uring/README.md still describes the 2048-entry geometry

The "Parallel test runs" section says unit tests shrink from "the production 256/2048 geometry" (line 159). It also says "A production-geometry worker takes 52 KiB" (line 167). Both now overstate the production ring. With this PR it is 256/1024, and the locked-memory footprint is about 36 KiB: 5 pages of rings plus 4 pages of SQEs, instead of 9 + 4. Anyone working out how many workers fit under RLIMIT_MEMLOCK would get it wrong.

Technical details
# Stale production ring geometry in moq-uring README

## Affected sites
- rs/moq-uring/README.md:159 — "256/2048 geometry" should be 256/1024
- rs/moq-uring/README.md:167 — "A production-geometry worker takes 52 KiB" should be ~36 KiB (rings 17,472 B → 5 pages, SQEs 256×64 = 16 KiB → 4 pages)

## Required outcome
- README numbers match `PRODUCTION_CQ_ENTRIES = 1024` / `PRODUCTION_SQ_ENTRIES = 256`.

Pullfrog  | Fix it ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The production completion queue limit decreases from 2048 to 1024, and the default transmit-buffer ceiling decreases from 1024 to 512. A new test checks that the estimated ring region is at most 32 KiB. The ENOMEM documentation and error message now describe memory-cgroup exhaustion, fragmented-host allocation, and the Linux 6.14-to-7.3 RLIMIT_MEMLOCK case. The quest notes now report the queue-size change and remove the prior investigation document.

Severity of issue fixed: Low

Merge Risk: Minimal · up to fe6e2

This change shrinks io_uring ring sizes and the default transmit buffer limit to avoid setup failures on fragmented hosts. No concrete merge-blocking risk was found. The only caveat is that the throughput effect of the smaller transmit ceiling was not measured.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: keeping the io_uring rings region within order 3 so setup survives fragmentation. The changeset reduces the production CQ from 2048 to 1024 entries, which matches this…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is specific and on-topic. It explains the io_uring ENOMEM failure in moq-relay, the cause (order-4 allocation for a 33,856 B rings region), and the changes: CQ reduced from 2048 to 102…
Full details: Title check

Explanation

The title names the main change: keeping the io_uring rings region within order 3 so setup survives fragmentation. The changeset reduces the production CQ from 2048 to 1024 entries, which matches this goal, and the title uses a conventional fix(...) prefix. It is specific and accurate, so it passes.

Full details: Description check

Explanation

The description is specific and on-topic. It explains the io_uring ENOMEM failure in moq-relay, the cause (order-4 allocation for a 33,856 B rings region), and the changes: CQ reduced from 2048 to 1024, udp tx_buffers_max reduced from 1024 to 512, the Error::ring message update, and removal of the finished quest. It also lists validation runs and the decisions behind the change. This matches the changeset in worker.rs, udp.rs, error.rs, and the quest README files.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a3f3b1f3-7c21-41ed-9658-eabfde28fa87
Commits

Reviewing files that changed from the base of the PR and between dcf9029 and fe6e2db.

Files selected for processing: 5

  • quest/m0/test-flakes-2/README.md
  • quest/m0/test-flakes-2/uring-setup-enomem.md
  • rs/moq-uring/src/error.rs
  • rs/moq-uring/src/udp.rs
  • rs/moq-uring/src/worker.rs
Files with no reviewable changes: 1

  • quest/m0/test-flakes-2/uring-setup-enomem.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

…ts 36 KiB

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated disabled auto-merge November 14, 2026 20:40
@kixelated

Copy link
Copy Markdown
Collaborator Author

@pullfrog review this pull request and submit an approving review if it is ready to merge

Your note on fe6e2db is fixed in 2a16a28: rs/moq-uring/README.md now says the production geometry is 256/1024 and a production worker takes 36 KiB (17,472 B of rings in 5 pages plus 16 KiB of SQEs in 4).

(Written by Claude Opus 5.5)

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes — the delta since the prior Pullfrog review (fe6e2db → 2a16a28).

  • Corrected README geometry — rs/moq-uring/README.md now states the 256/1024 production geometry and 36 KiB per production worker, which resolves the stale 256/2048 and 52 KiB figures flagged earlier.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The stale README note from the prior review is fixed.

Reviewed changes

I reviewed what changed since the prior Pullfrog review at fe6e2db: one docs commit.

  • README geometry corrected: rs/moq-uring/README.md now gives the production geometry as 256/1024 and says a production worker takes 36 KiB. That matches PRODUCTION_SQ_ENTRIES = 256 and PRODUCTION_CQ_ENTRIES = 1024. The rings come to 17,472 B, which rounds up to 5 pages, and the SQEs take 256 × 64 B = 16 KiB, which is 4 pages. Together that is 9 pages, or 36 KiB. The 20 KiB unit-test figure still works out the same way: 8,768 B of rings in 3 pages plus 8 KiB of SQEs in 2 pages.

A grep over rs/moq-uring, rs/moq-relay, and doc/ found no other mention of the old 2048-entry CQ or the 52 KiB figure.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 14, 2026
Merged via the queue into main with commit dcad5bf Nov 14, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m0/test-flakes-2/uring-setup-enomem branch November 14, 2026 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant