Skip to content

pipes: AdvanceTo on a reader holding one recv computes its offsets from the positions themselves, without three GetOffset calls - #400

Merged
MDA2AV merged 2 commits into
mainfrom
fix/pipe-advance-single-segment
Oct 25, 2026
Merged

MDA2AV merged 2 commits into
mainfrom
fix/pipe-advance-single-segment

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

TcpConnectionPipeReader.AdvanceTo turned both positions into byte offsets with three ReadOnlySequence.GetOffset calls: one for the sequence start, then one each for consumed and examined. It rebases on the start because GetOffset counts from the start segment, not from where the sequence begins. Each call bounds-checks and, across segments, walks the chain.

Most reads hand out one recv slice, and then none of that is needed. A position into the slice is the slice and an index into its memory, so the offset is the index less what was already consumed.

What changes:

  • Fast path: one slice held, both positions on it, the last result's sequence starting on it, and both indexes inside what the slice still holds. The bound is the slice's own unconsumed length, the one GetOffset's bounds check uses. Offsets are then computed from the indexes directly.
  • Everything else: more than one slice, a position on another segment, an index out of range, a sequence that isn't the last result's. These take the existing GetOffset path unchanged, which also keeps refusing an index the sequence doesn't hold.

After a read, the two paths compute the same offsets and refuse the same positions. They differ only on an AdvanceTo with no read since the last result, which the PipeReader contract forbids (Pipe throws for it):

  • Main measures it from the stale sequence's start and over-counts. Its counters go negative, and a later read parks on bytes that already arrived.
  • This PR measures from what is consumed now, and its counters stay right.

From review

An adversarial review of the first commit tested the equivalence with a harness. The harness compiles main's AdvanceTo and this one verbatim, runs both from the same states, and compares exceptions, offsets and the state they leave. The equivalence held after every read. The review found five smaller things, all fixed in the second commit:

  • The bound was _heldBytes. That equals the slice's unconsumed length only while the counters are right. Once two contract violations had made them negative, any index on the slice passed, and a read could then return empty results forever. The bound is now the slice's length less what is consumed, so it holds whatever the counters say.
  • A stale _lastSequence. The overflow error paths and ReleaseHeld leave one. The fast path now applies only when the last result's sequence starts on the head slice.
  • The comment claimed GetOffset refuses a position on another segment. For a one-segment sequence it doesn't, on main either.
  • Wrapping. The casts that turn a negative offset into an out-of-range one are now explicitly unchecked.
  • Tests. The test checked only the consumed bound. It now checks examined too, and a new test covers a read holding two recvs (below).

The same harness on the second commit, with two seeds:

inputs calls per seed through the fast path differ from main
after a read, counters intact (1.46M exhaustive pairs among them) 4.93M 975K 0
after a read, counters already broken by an earlier contract violation 477K 72K 0, where the first commit differed in 1,621 of 187K
no read since the last result 518K 23K 21.5K

In that last row, this PR never leaves the counters wrong where main's stay right. Main's go wrong where these stay right in about 10K calls per seed.

Measured

Per AdvanceTo, with a micro-benchmark of the offset step pinned to one P-core: one recv slice, partly consumed, the caller consuming everything held, 10 alternating rounds of 50M each.

  • Skipped: the three GetOffset calls, which take 7.09 ns.
  • Instead: the second commit's compares cost 0.70 ns in the loop (the first commit's: 0.59 ns). Loads hoist out of a loop like that, so the figure flatters them.

End to end, it is below what this box resolves. These figures are for the first commit; the second adds a reference compare and a length load to the fast path. wrk on Tcp/Pipe, measured as in the pipe-adapter PR (#399):

  • Setup: 2 reactors on two P-cores, wrk -t4 -c64 on other cores.
  • Order: 12 rounds, each running every build back to back in rotated order.
  • Figures: the median of the per-round ratios against main.
user CPU/req CPU/req req/s
Tcp/Pipe −1.2% [−9.6..+2.5] −0.6% +0.8%
Tcp/Raw (doesn't use the pipe reader) −1.9% [−7.6..+5.0] +1.5% −1.5%
control, main vs main, on Tcp/Pipe +0.6% [−4.7..+10.1] +0.0% +0.2%

Seven nanoseconds is about 4% of the 164 ns of user-mode CPU a Pipe request costs here, and under 0.4% of its total CPU. The Raw row, which this change cannot affect, moves as much as the Pipe one.

Tests

In PipeReaderContractTests (E2E):

  • pipereader: AdvanceTo refuses a position outside what it handed out, past a recv's end and inside its consumed part
    • What it does: a request arrives in one recv. The test hands AdvanceTo a position one past the recv's end, as consumed and as examined, then consumes the header and hands it a position inside the consumed header. Every one must be refused, and the body must still read whole afterwards.
    • Vacuity guard: it asserts the request arrived in one recv, so the positions really take the new path.
    • Clear failures: an accepted position is reported at once, because the byte counters are wrong from then on and the next read would park on bytes that already arrived.
  • pipereader: AdvanceTo refuses a position past the first recv while the reader holds two, new in the second commit
    • What it does: two sends, 300 ms apart, so the reader holds two recvs. A position one past the first recv's end must be refused, and all five bytes must still be there. A bound over everything held would accept it, and consume bytes of the second recv it never pointed at.
    • Vacuity guard: it fails, rather than passing, if the two sends arrive as one recv.

Red-checked, one break per variant:

result
no range checks in the fast path (first commit) FAIL: got [past the recv's end: accepted]
upper bound only, a signed compare (first commit) FAIL: got [inside its consumed part: accepted]
no examined range check FAIL: got [examined past the recv's end: accepted]
bound by everything held, without the one-slice check FAIL: got [past the first recv: accepted]
this PR PASS

The rest of AdvanceTo is covered as before:

  • PipeReaderContractTests: examined-to-the-end, a partial consume of one recv, and buffers returning as consumption passes them.
  • RecvBufferReclaimTests.
  • Every h2c and kTLS test that reads through the pipe.

All suites pass: Unit 105, E2E 328, Http 89, Tls 194, Chaos 80, File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending).

  • The tls module was loaded, so no Tls case skipped.
  • The Pg and Redis skips need servers this box doesn't run.

Merges cleanly with the pipe-adapter change opened alongside it (#399), with its follow-up (#401), and with #386, #384 and #363. Against #307 the only conflict is in HardeningTests.cs, which #307 already has with main.

API: none.

🤖 Generated with Claude Code

…om the positions themselves, without three GetOffset calls

TcpConnectionPipeReader.AdvanceTo turned both positions into byte offsets with three
ReadOnlySequence.GetOffset calls - the sequence start, consumed and examined - rebasing on the start
because GetOffset counts from the start segment. Each call bounds-checks, and across segments walks
the chain.

Most reads hand out one recv slice, and a position into it is that slice and an index into its
memory, so the offset is the index less what was already consumed. AdvanceTo now does that when one
slice is held, both positions are on it and both indexes are inside what it still holds. Anything
else - more slices, a position on another segment, an index out of range - takes the GetOffset path
as before, which also keeps refusing a position the sequence does not hold.

A micro-benchmark of the offset step (one recv slice, partly consumed, everything consumed) puts the
three GetOffset calls at 6.9-7.3 ns, which is what this skips. End to end that is below what this
box resolves: wrk on Tcp/Pipe, 12 paired rounds against main, user CPU per request -1.2%, against
-1.9% on Tcp/Raw, which does not use the pipe reader, and +0.6% for the main-vs-main control.

Test (E2E, PipeReaderContractTests): AdvanceTo refuses a position outside what it handed out, past a
recv's end and inside its consumed part, and the body stays readable after both refusals; it
asserts the request arrived in one recv, so the positions take the new path, and reports an
accepted position at once, since the counters are wrong from then on and the next read would park.
Red-checked: with no range checks in the fast path it fails with "past the recv's end: accepted",
and with an upper bound only (a signed compare) with "inside its consumed part: accepted"; the
fast path as committed passes.
… unconsumed bytes, and applies only to the last result's sequence

From review of the first commit. Its bound was _heldBytes, which equals the one recv's unconsumed
length only while the counters are right; once a caller had broken them (two AdvanceTo for one
read, twice) any index on the recv passed. The bound is now the recv's own unconsumed length, the
one GetOffset's bounds check uses, so it holds whatever the counters say. And the path applies only
when the last result's sequence starts at that recv, so a stale or reset sequence - the overflow
error paths and ReleaseHeld leave one - takes the GetOffset path as before. The comment no longer
claims GetOffset refuses a position on another segment, which for one segment it does not; the
casts that turn a negative offset into an out-of-range one are explicit about wrapping.

The test now also refuses an examined position past the recv's end, and a new one refuses a
position past the first recv while the reader holds two, which a bound over everything held would
accept.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant