Skip to content

tcp: the sweep finds a reset that io_uring never delivered, so a read stops waiting on a peer that is gone - #408

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix/403-reset-after-partial
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix/403-reset-after-partial

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 25, 2026

Copy link
Copy Markdown
Owner

Fixes #403.

Cause: a kernel bug, not the reactor. A peer that sends its last bytes and resets before the reactor's task work runs gets both wakeups handled in one pass of io_uring's poll loop. One recv takes the bytes, the loop drops both wakeups with it, and with nothing left to read the multishot recv waits for an event that never comes.

  • Upstream: fixed by a68ed2df7213, "io_uring/poll: fix multishot recv missing EOF on wakeup race" (7.0-rc5, backported to the stable trees; reported as liburing#1549 for AF_UNIX). This box's 6.14.0-37 doesn't have it.
  • Why only a reset: after a FIN the socket still reports a byte to read, so io_uring runs the recv again and it returns the EOF. A reset leaves only its error. That matches the issue's table, where only "part of a request, then RST" stuck.
  • Evidence: the stuck sockets are in state CLOSE with ECONNRESET still in SO_ERROR. A recv after the reset would have consumed that error, so none ran. The kernel never posted the completion, and the reactor dropped nothing.
  • Rate: it depends on how far behind the reactor's task work runs. 1,044 of 1,048 connections stuck on a server still JIT-ing, 149 of 42,192 (0.35%) on a warm one.

The workaround: the TCP sweep asks a socket whose wait on the peer has just passed one tick (250 ms) whether it hung up. That is one poll() that doesn't wait, once per wait. POLLHUP ends the connection the way the read timeout does: TcpSweepClose's shutdown() wakes the stuck recv, which reports the error and runs the usual teardown. poll() leaves the error in place for that recv.

  • Once per wait: the sweep keeps the previous sweep's time, so a wait passes the one-tick mark between exactly one pair of sweeps, however late they run. No per-connection state.
  • Cost: nothing on the request path. A busy connection never waits a tick. An idle keep-alive connection costs one poll() each time it goes quiet for longer than a tick. On a kernel with the fix the check never finds anything.
  • Found: 250-500 ms after the read parks, instead of at ReadTimeoutMs (60 s by default), or never with the clock off.
  • Not covered: a connection off the read clock (DisableReadTimeout, which only the Kestrel adapter uses), since WaitingOnPeer leaves it out. Kestrel's own timeouts bound it.

Tests

New, in E2E TcpTimeoutTests:

  • tcp/reset: a reset that comes in with the peer's last bytes ends the read, with the clock off (#403)
    • What it does: eight clients send their last bytes and reset while the reactor is held (ScheduleOnReactor). The ring uses DEFER_TASKRUN, so each connection's bytes and reset land in one task-work pass: the race, made deterministic.
    • Vacuity guard: every read must have taken those last bytes before it ended.
  • tcp/reset: control: a FIN that comes in with the peer's last bytes ends the read

Red-checked by compiling the same test file against main's core, in a separate build:

reset test FIN control
main FAIL in 3 of 3 runs: 0 of 8 reads ended within 4 s PASS
this PR PASS in 4 of 4 PASS

On a kernel with the upstream fix the reset test passes on main too. There it guards the behaviour, not the workaround.

The issue's repro shape: 1 reactor, a 1,024 × 4 KB ring, ReadTimeoutMs 0, one client connecting in a loop for 1 s, sending part of a request and then an RST.

connections handlers still parked accepted sockets still open
main, warm server 42,577 85 85
this PR, warm 42,918 0 0
main, server still JIT-ing 1,048 1,044
this PR, server still JIT-ing 1,112 0
main, incremental buffers 42,815 10 10
this PR, incremental buffers 43,177 0 0

All suites pass: Unit 121, E2E 330, Http 89, Tls 194 (the kTLS cases included), Chaos 80, File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending). The Pg and Redis skips need servers this box doesn't run.

Bench: wrk on Tcp/Pipe: 2 reactors on 2 CPUs, wrk -t4 -c64, a 2-byte body. Six rounds ran main, a second main worktree as the control, and this PR back to back, in rotated order. The median per-round ratio against main is -0.2% for this PR and -0.5% for the control. CPU per request: 2.753 µs on main, 2.751 on the control, 2.754 here. A busy connection never waits a tick, so the check never runs in this load.

Merges cleanly with #407, #406, #393, #386, #384 and #363. Against #307 the only conflict is in HardeningTests.cs, which #307 already has with main.

API: none. Native.poll and pollfd are internal.

🤖 Generated with Claude Code

… stops waiting on a peer that is gone

A peer that sends its last bytes and resets before the reactor's task work runs gets both wakeups
handled in one pass of io_uring's poll loop. One recv takes the bytes, the loop drops both wakeups
with it, and with nothing left to read the multishot recv waits for an event that never comes. That
is the kernel bug fixed by a68ed2df7213 ("io_uring/poll: fix multishot recv missing EOF on wakeup
race", in 7.0 and the stable trees); this box's 6.14 does not have it. A FIN escapes it, because the
socket still reports a byte to read and the recv runs again. A reset leaves only its error. So the
read waited on a peer that was gone until ReadTimeoutMs, holding its recv buffer, and forever with
the clock off (#403). The stuck sockets were in state CLOSE with ECONNRESET still in SO_ERROR: no
recv had run on them since the reset.

The sweep now asks a socket whose wait on the peer has just passed one tick whether it hung up: one
poll() that does not wait, once per wait. POLLHUP ends the connection the way the read timeout does.
Its shutdown() wakes the stuck recv, which then reports the error and runs the usual teardown. The
request path is unchanged, and a connection that is busy never waits a tick.

Tests: "tcp/reset: a reset that comes in with the peer's last bytes ends the read, with the clock
off (#403)", and its control for a FIN (E2E, TcpTimeoutTests). Eight clients send their last bytes
and hang up while the reactor is held, so each connection's bytes and hang-up land in one task-work
pass. Red-checked by compiling the same test file against main: 0 of 8 reads ended, in three runs
out of three. With the fix all 8 end, in four runs. The control passes on both.
@MDA2AV

MDA2AV commented Oct 26, 2026

Copy link
Copy Markdown
Owner Author

Closing: the cause is a kernel bug (fixed upstream by a68ed2df7213), and ReadTimeoutMs already bounds it, the same way it bounds a client that sends part of a request and goes quiet. The root cause is written up on #403.

@MDA2AV MDA2AV closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tcp: a peer that sends part of a request and then resets is sometimes never noticed, and its handler holds a recv buffer until ReadTimeoutMs

1 participant