Repository navigation
Conversation
…, four times RingEntries by default Ring.Create set no IORING_SETUP_CQSIZE, so the CQ was the kernel's default, twice the SQ, though a reactor's completions are not bounded by its submissions: a multishot recv or accept posts many from one, and the batch being dispatched holds its CQ slots until the pass ends, so what the pass itself submits and the kernel completes on the spot lands behind it. A completion with no room waits in an allocated kernel list, and a multishot recv or accept whose completion goes there ends. CqEntries 0 (the default) is four times RingEntries, capped at the kernel's 65536, so no RingEntries gets a smaller CQ than before; an explicit value below RingEntries or above 65536 is refused at construction, by name. It costs 16 bytes an entry, charged like the rest of the ring: on 6.14 the ring at the defaults takes 1028 KB of RLIMIT_MEMLOCK instead of 772 KB (README updated). Tests (Unit, CompletionQueueTests): a handler on a 16-entry SQ sends 48 single bytes in one pass; with the default CQ nothing overflows, with CqEntries 32, the old CQ, it does. Confirmed both ways: with the reactor ignoring CqEntries the default test fails. HardeningTests' full-CQ test sets CqEntries 8, the CQ RingEntries 4 used to give it.
This was referenced Oct 23, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #95.
The CQ was always the kernel's default, twice the SQ: 16384 entries at the default
RingEntriesof 8192. Nothing ties a reactor's completions to its submissions. A multishot recv or accept posts many from one submission. And the CQ keeps the batch being dispatched until the pass ends, so whatever the pass itself submits, and the kernel completes on the spot, lands behind it. A completion with no room waits in an allocated kernel list until the next enter, and a multishot recv or accept whose completion goes there ends and has to be re-armed.ServerConfig.CqEntriesnow sets the CQ (IORING_SETUP_CQSIZE). The default, 0, is four timesRingEntries, capped at the kernel's 65536, so noRingEntriesgets a smaller CQ than before. An explicit value belowRingEntriesor above 65536 is refused at construction, by name, like the other sizes.What can overflow depends on the kernel:
io_uring_enterruns at most 20 deferred completions. Measured here on 6.14: 2,000 and 20,000 armed recvs, all made ready at once, came out 20 per call. A burst of network events can't fill the CQ there. What can is a pass whose own submissions complete on the spot (sends with room in the socket buffer, page-cache reads) in greater numbers than the CQ has left behind the batch.Cost: 16 bytes an entry, charged like the rest of the ring, which on 6.14 means
RLIMIT_MEMLOCK. At the defaults the ring goes from 772 KB to 1028 KB (measured by counting the rings that fit under twoRLIMIT_MEMLOCKvalues: 771 KiB a ring before, 1028 KiB now, and 1542 KiB at a 65536-entry CQ). With its buffer ring a reactor is about 1.1 MB, so an 8 MBulimit -lfits seven instead of ten. The README says so.Not done: the issue's second suggestion, surfacing the mapped
cq_off.overflowcounter. It stayed 0 on 6.14 while 16 completions sat in the overflow list, because it counts only completions dropped for lack of memory, so it would always read 0. The live signal isIORING_SQ_CQ_OVERFLOW, already read asReactor.CqOverflowedby the kTLS handoff.Tests
New, in
CompletionQueueTests(Unit, whereReactor.CqOverflowedis visible). A handler on a 16-entry SQ sends 48 single bytes in one pass. Every send completes as it is submitted, each time the SQ fills, behind the batch being dispatched.ring: by default a pass that completes three times RingEntries fits the CQring: control: the kernel's CQ, twice the SQ, overflows on the same pass, which proves the first one can fail.CqEntries(the old CQ)CqEntries = 32Changed:
HardeningTests' "an incremental recv ended by a full completion queue still hands over the peer's bytes" now setsCqEntries = 8. It needs an 8-entry CQ, whichRingEntries = 4used to imply and now would make 16.RingSetupMessageTestsrefusesCqEntriesbelowRingEntriesand above 65536, and its control constructs at 65536.API: adds
ServerConfig.CqEntriesand aRing.Create(uint, uint)overload.Ring.Create(uint)keeps the kernel's default CQ. Nothing is removed or changed.All suites pass: Unit 106, E2E 311, Chaos 80, Http 88, Tls 169 (+14 skipped), File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending). The skips need servers this box doesn't run.
Bench
Flat. Setup:
wrk -t4 -c64on other cores.origin/main.Tcp/RawTcp/Raw's handler, resumed on the thread pool after each read