[MDA2AV]

#399 and #400 are merged, so this is now one commit on top of main, 200dc43.

After #399 a read could wait two ways: the pipe reader through TryReadOrPark, and ReadAsync through its own copy of the arm protocol. A flush signalled either an adapter or the connection's own source, chosen by a null check. Now there is one way for each:

• The connection is its own listener. TcpConnection implements IRecvListener and IFlushListener. ReadAsync is TryReadOrPark(this), and FlushAsync / FlushIfIdleAsync arm through the same ArmFlush as TryFlushOrPark. Its OnParking readies its own value task, and its OnRecvReady / OnFlushDone complete it.
• One read slot. _readWaiter replaces the _armed flag and the separate listener field.
• ResetRead is a no-op. A read readies its own value task when it parks. Callers written to the old contract keep working, including any that call ResetRead while a read is parked, which used to throw that read's continuation away.

The raw API stays as it is; only its insides changed. The public API is identical to main's (a reflection dump of the assembly, 900 lines).

How a read parks

1. Claim: a CompareExchange takes the slot with a mark no completion delivers to. A read racing it is refused having touched nothing, including the listener's value task, which two reads through one adapter (or two of ReadAsync's) share.
2. Ready: the listener readies its value task (OnParking). A re-check after the claim catches a completion that landed just before it.
3. Publish: a second CompareExchange puts the listener in the slot. A completion that came while the listener was readying marks the claim instead, and the read serves what was queued rather than parking.

A completion takes a published park back with one CompareExchange, and only the side that takes it delivers.

A flush was already safe on ownership: it takes the in-progress guard first, then the listener readies, then the arm. What changes is the end: the completion now disarms the flush and takes its listener before letting go of the guard.

What it fixes

On main:

• ReadAsync's double delivery. Its race recovery took the arm back without looking at the result. So when a recv had already taken the arm, both delivered: the caller was served at once, and the recv also completed the read's value task. That stale completion raced the caller's ResetRead and next read, with three outcomes:

  • the value task came out torn;
  • the next read, parked for real, looked complete, and the one after it was refused as already armed;
  • the next recv's SetResult threw on the reactor.

  The concurrency review of pipes: the TCP pipe reader and writer park on the connection itself, so a read or flush that waits has one value-task source, not two #399 hit it 403 times in 400,000 rounds. Run on main itself (1fe45aa), this PR's raw-read race test broke 5 and 53 of 200,000 rounds, and in one of the two runs the thread playing the reactor threw from SetResult. It reaches every raw reader: TcpConnectionStream, the userspace-TLS pipes, and the TLS handshake. On main that includes the pipe reader too, which is GenHTTP's path. The stress run behind pipes: the TCP pipe reader and writer park on the connection itself, so a read or flush that waits has one value-task source, not two #399 hit "ReadAsync already armed" there under thread hops.

• ResetRead during a parked read reset the value task the read was parked on, and threw its continuation away. On main, the test's parked read is never resumed.

• CompleteFlush let go of the in-progress guard before it disarmed the flush. A flush that took the guard in that gap reset the value task the completing flush's caller was waiting on. On main itself, two runs of 200,000 rounds each:

  • the completing flush's caller was never resumed, in 95 and 40 rounds;
  • the completion's signal landed on the next flush instead, whose own completion then signalled a second time and threw, in 83 and 15 rounds. On the reactor thread, CompleteFlush's own comment says that throw takes the process down.

  It takes a second flush starting while the first one completes, meaning writers that aren't serialized, which GenHTTP's websocket writes can be (tcp, tls: a second writer on a connection is not refused, and can corrupt memory #397). With pipes: the TCP pipe reader and writer park on the connection itself, so a read or flush that waits has one value-task source, not two #399's listeners, that flush could also be refused as already armed, or signalled in place of the one completing.

From #399, and so on main since it merged, when two reads park at once:

• The refused read stored its listener before the exchange that refused it, so the parked read's delivery could go to it instead.
• It also readied its listener before that exchange, so two reads through one adapter could reset the value task the parked read's caller waits on.

Main before #399 had neither: the pipe reader parked through ReadAsync, whose arm exchange came first.

Measured

wrk as in #399:

• Setup: 2 reactors on two P-cores, wrk -t4 -c64 on other cores, 2-byte body.
• Order: every round runs each build back to back, in rotated order.
• Figures: the median of the per-round ratios, with their range, over 10 s rounds.
• User CPU: the server's user-mode time per request, from /proc/PID/stat. The kernel's TCP path is 92–95% of a request's CPU here.

Run 1 had 12 rounds and five builds: main as it was before #399 and #400 (1fe45aa), #399, this PR, the variant that claims and publishes with one CompareExchange, and main again as the noise control.

user CPU/req	Tcp/Pipe	Tcp/Raw
#399 vs main	−13.4% [−17.5..−7.7]	−3.8% [−13.2..+8.4]
this PR vs main	−10.2% [−14.6..−4.9]	−4.9% [−12.0..+9.5]
this PR vs #399	+1.4% [−2.8..+14.6]	+0.5% [−7.8..+7.1]
this PR vs the one-CompareExchange variant	−1.6% [−7.6..+6.2]	+3.0% [−7.8..+14.0]
control: main vs main	+0.3% [−4.2..+7.1]	−1.5% [−9.6..+6.9]

Run 2 had 16 rounds, Tcp/Pipe only, with #399 twice. This PR vs #399 was +1.4% [−6.1..+10.1], and #399 vs itself −2.1% [−6.1..+6.6].

What that says:

• Pipe vs pipes: the TCP pipe reader and writer park on the connection itself, so a read or flush that waits has one value-task source, not two #399: user CPU may be a nanosecond or three higher. It was +1.4% in both runs (0.142 → 0.145 µs), higher in 19 of 28 rounds, while the same binary spreads a few percent either way. CPU per request and req/s don't move: +0.6% and −0.8% in run 1, −0.2% and +0.3% in run 2.
• Raw: its reads now park through the slot as well, and it doesn't move.
• The claim's second CompareExchange: it doesn't show.

Tests

New, in WaiterSlotTests (Unit). Five are races, each 200,000 rounds with a thread of the test's own playing the reactor's part. Each asserts that at least 1,000 rounds overlapped, so a run where the race never happened can't pass; this PR's runs overlap in about 22,000 to 142,000.

• conn: a raw read racing its recv completions leaves the reads after it working
• conn: two reads parking at once cannot take each other's place
• conn: a read refused for racing a park of the same listener never readies that listener
• conn: two raw reads at once: the one refused leaves the parked one's value task to complete
• conn: a flush that starts while the last one's completion is delivering is neither refused as armed nor signalled in its place

Two are deterministic:

• conn: ResetRead while a raw read is parked leaves the read to complete
• conn: a completion that lands while a read readies its listener is served by that read, which then lets go of the slot. The completion is a wake from inside the listener's OnParking, which is exactly the gap between the claim and the publish.

Red-checked, two runs per variant:

test	#399's connection code, now main's	one CompareExchange claiming and publishing, after the listener readies	this PR (3 runs)
raw read racing its recv completions	FAIL: value task torn, once thrown on the side playing the reactor	PASS	PASS
ResetRead while a raw read is parked	FAIL	PASS	PASS
two reads parking at once	FAIL: 15,937 / 15,452 rounds	PASS	PASS
a read racing a park of the same listener	FAIL: 15,574 / 17,110	FAIL: 9,907 / 19,176	PASS
two raw reads at once	FAIL: every round, as ReadAsync there leaves readying its value task to ResetRead	FAIL: 5,567 / 11,120	PASS
a flush starting while the last one's completion delivers	FAIL: 35 / 13	PASS	PASS
a completion landing while a read readies	PASS	PASS	PASS

The last test guards the claim's own path, which the other two variants don't have. Two breaks of that path, one at a time:

• A completion leaves a claim unmarked: the read parks with the wake already spent. Two of pipes: the TCP pipe reader and writer park on the connection itself, so a read or flush that waits has one value-task source, not two #399's race tests fail with it.
• A read served after a mark keeps the slot: the next read is refused as already armed. The raw-read race fails with it, in 316 rounds.

With a counter added, the Unit suite's races take the mark path 1,000–1,350 times per run, and the claim's re-check 9,000–18,000.

Docs: the connection section of dev-core.

All suites pass on top of main, #400 included: Unit 121, E2E 328, Http 89, Tls 194, Chaos 80, File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending).

• The tls module was loaded, so no Tls case skipped.
• The Pg and Redis skips need servers this box doesn't run.

Merges cleanly with #393, #386, #384 and #363. Against #307 the only conflict is in HardeningTests.cs, which #307 already has with main.

🤖 Generated with Claude Code