Repository navigation
reactor: handoffs from other threads share one eventfd write per loop pass, as posts already did - #387
Merged
Conversation
… pass, as posts already did Only the post queue coalesced its wakes. A buffer return, a flush, a recycle or a client op handed over from another thread each wrote the eventfd, a syscall per item for a handler that hops threads. The post queue's flag is now the reactor's (_wakePending): every handoff wakes through Wake(), which writes only when it is the one to set the flag, and each loop pass clears it first, before any queue is drained - the exchange DrainPostQ made since #321, moved to the top of the pass, so the loop still pays one per pass. Stop() still writes unconditionally: it must wake a parked loop whatever the flag says. Test (AffinityTests): on a reactor that never runs, whose wake is the test's own eventfd, 100 of each of the six handoffs write it once, and after a pass's clear the next ones write it once more. Confirmed both ways: with the five other queues writing directly again, the 600 handoffs make 501 writes. The post-vs-drain race test now drains the way a pass does: ClearWakePending, then DrainPostQ.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #96.
Only the post queue coalesced its eventfd wakes. Every other handoff from another thread wrote the eventfd once per item, a
write(2)apiece:EnqueueReturnQ,EnqueueReturnQIncremental)EnqueueFlush)EnqueueRecycle)HandOff)A handler that stays on the reactor never reaches any of them. One that leaves it, such as code after a
ConfigureAwait(false)await or a ring client called from the thread pool, pays one per return, flush and op.The post queue's flag is now the reactor's,
_wakePending. Every handoff ends inWake(), which writes the eventfd only when it is the one to set the flag. Each loop pass clears the flag first, before any queue is drained. That is the sameInterlocked.Exchangethat #321 put inDrainPostQ, moved earlier in the pass, so the reactor still pays one locked exchange per pass and nothing more. The ordering argument is unchanged from #321: every drain now comes after the clear, so a handoff that lands mid-drain either is drained or sets the flag again and wakes the next pass.Stop()still writes unconditionally, because it has to wake a parked loop whatever the flag says.No new state: the post flag is renamed and moved. The diff in the core is +25/−13.
Tests
New, in
AffinityTests:affinity: handoffs from another thread share one eventfd write until the loop's next passIt uses a reactor that never runs, whose wake is an eventfd of the test's own, so the counter it reads back is the number of writes. The test makes 100 of each of the six handoffs (return, incremental return, flush, recycle, client op, post), then a clear as a pass would, then two more handoffs.
Changed: the #321 race test drains the way a pass now does,
ClearWakePendingand thenDrainPostQ, and reaches the flag under its new name.No public API change.
All suites pass: Unit 102, E2E 312, Chaos 80, Http 88, Tls 169 (+14 skipped), File 5, Pg 7 (+11 skipped), Redis (6 skipped, 1 pending). The skips need servers this box doesn't run.
Measured
Setup:
wrk -t4 -c64on other cores.origin/main.write(2)per request comes from/proc/PID/io(syscw). The eventfd wakes are the only writes these servers make.write(2)/req, main → this PRTcp/RawTcp/Hop(posts, already coalesced)Every round of the off-reactor workload agreed, at +6.5% to +10.7% req/s. Its handler is
Tcp/Raw's, except that after each read it resumes on the thread pool and stays there. Its buffer return and its flush therefore go throughEnqueueReturnQandEnqueueFlush, the path a handler takes when it awaits library code withConfigureAwait(false):