Repository navigation
Conversation
… stops waiting on a peer that is gone
A peer that sends its last bytes and resets before the reactor's task work runs gets both wakeups
handled in one pass of io_uring's poll loop. One recv takes the bytes, the loop drops both wakeups
with it, and with nothing left to read the multishot recv waits for an event that never comes. That
is the kernel bug fixed by a68ed2df7213 ("io_uring/poll: fix multishot recv missing EOF on wakeup
race", in 7.0 and the stable trees); this box's 6.14 does not have it. A FIN escapes it, because the
socket still reports a byte to read and the recv runs again. A reset leaves only its error. So the
read waited on a peer that was gone until ReadTimeoutMs, holding its recv buffer, and forever with
the clock off (#403). The stuck sockets were in state CLOSE with ECONNRESET still in SO_ERROR: no
recv had run on them since the reset.
The sweep now asks a socket whose wait on the peer has just passed one tick whether it hung up: one
poll() that does not wait, once per wait. POLLHUP ends the connection the way the read timeout does.
Its shutdown() wakes the stuck recv, which then reports the error and runs the usual teardown. The
request path is unchanged, and a connection that is busy never waits a tick.
Tests: "tcp/reset: a reset that comes in with the peer's last bytes ends the read, with the clock
off (#403)", and its control for a FIN (E2E, TcpTimeoutTests). Eight clients send their last bytes
and hang up while the reactor is held, so each connection's bytes and hang-up land in one task-work
pass. Red-checked by compiling the same test file against main: 0 of 8 reads ended, in three runs
out of three. With the fix all 8 end, in four runs. The control passes on both.
This was referenced Oct 26, 2026
Owner
Author
|
Closing: the cause is a kernel bug (fixed upstream by a68ed2df7213), and |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #403.
Cause: a kernel bug, not the reactor. A peer that sends its last bytes and resets before the reactor's task work runs gets both wakeups handled in one pass of io_uring's poll loop. One recv takes the bytes, the loop drops both wakeups with it, and with nothing left to read the multishot recv waits for an event that never comes.
ECONNRESETstill inSO_ERROR. A recv after the reset would have consumed that error, so none ran. The kernel never posted the completion, and the reactor dropped nothing.The workaround: the TCP sweep asks a socket whose wait on the peer has just passed one tick (250 ms) whether it hung up. That is one
poll()that doesn't wait, once per wait.POLLHUPends the connection the way the read timeout does:TcpSweepClose'sshutdown()wakes the stuck recv, which reports the error and runs the usual teardown.poll()leaves the error in place for that recv.poll()each time it goes quiet for longer than a tick. On a kernel with the fix the check never finds anything.ReadTimeoutMs(60 s by default), or never with the clock off.DisableReadTimeout, which only the Kestrel adapter uses), sinceWaitingOnPeerleaves it out. Kestrel's own timeouts bound it.Tests
New, in E2E
TcpTimeoutTests:tcp/reset: a reset that comes in with the peer's last bytes ends the read, with the clock off (#403)ScheduleOnReactor). The ring uses DEFER_TASKRUN, so each connection's bytes and reset land in one task-work pass: the race, made deterministic.tcp/reset: control: a FIN that comes in with the peer's last bytes ends the readRed-checked by compiling the same test file against main's core, in a separate build:
On a kernel with the upstream fix the reset test passes on main too. There it guards the behaviour, not the workaround.
The issue's repro shape: 1 reactor, a 1,024 × 4 KB ring,
ReadTimeoutMs0, one client connecting in a loop for 1 s, sending part of a request and then an RST.All suites pass: Unit 121, E2E 330, Http 89, Tls 194 (the kTLS cases included), Chaos 80, File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending). The Pg and Redis skips need servers this box doesn't run.
Bench: wrk on
Tcp/Pipe: 2 reactors on 2 CPUs,wrk -t4 -c64, a 2-byte body. Six rounds ran main, a second main worktree as the control, and this PR back to back, in rotated order. The median per-round ratio against main is -0.2% for this PR and -0.5% for the control. CPU per request: 2.753 µs on main, 2.751 on the control, 2.754 here. A busy connection never waits a tick, so the check never runs in this load.Merges cleanly with #407, #406, #393, #386, #384 and #363. Against #307 the only conflict is in
HardeningTests.cs, which #307 already has with main.API: none.
Native.pollandpollfdare internal.🤖 Generated with Claude Code