Repository navigation
DDIR server: imports arrive as the producer's arrangement - #918
Merged
frankmcsherry merged 1 commit intoOct 12, 2026
Merged
Conversation
frankmcsherry
force-pushed
the
arranged-imports
branch
from
October 12, 2026 20:36
6d35466 to
a0d249a
Compare
frankmcsherry
changed the base branch from
master-next
to
ddir-share-arrangements
October 12, 2026 20:36
frankmcsherry
force-pushed
the
arranged-imports
branch
from
October 12, 2026 20:41
a0d249a to
4c750e9
Compare
frankmcsherry
changed the base branch from
ddir-share-arrangements
to
master-next
October 12, 2026 20:41
A program that imported a published trace received it as rows (or, in columnar mode, as containers), and each of its joins, reduces, and arranges of the import built a private copy of an arrangement the producer already maintains. Imports now arrive as that arrangement, entered into the program's scope (`TraceEnter`), on both backends. vec: `Arr` is a local or an imported arrangement. Join and reduce take either through the cursor tactics, as an entered batch's cursor lifts its times. corgi: times are integer lanes, and a host epoch `e` and the program time `(e, [])` have the same lanes, so an imported chunk is read at the program's time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and `RetimeReduce` wrap the proxy tactics and retime imported batches as they cross in, one `Rc` per chunk. Columnar exports, generated sources included, reach columnar programs this way. An imported arrangement also enters child scopes, iterative ones included, as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads as rows is flattened once per scope, and the rows enter its children with the arrangement. `Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the server binary, restores rows (containers in columnar mode). A shape ascription on an arranged import is checked by a side operator over the trace's batches, which produces nothing: on vec always, on corgi when the export's shape was not known at install (as TimelyDataflow#927 checks containers). examples/import_bench: one producer of 1M edges, then 8 consumers each joining the import twice, 1000 edges replaced per tick. Corgi without arranged imports reads containers (TimelyDataflow#927). vec corgi rows arranged containers arranged install-to-answer, ms ~400 ~3 27-42 ~1 memory per consumer, MiB ~95 ~0.5 ~54 ~0.3 tick, 8 consumers, ms 13.3 2.0 3.1 0.9 Consumers that join the import inside a loop (reachability) are alike. After 500 ticks of churn the published trace holds the same updates with 0 or 8 importers sharing it. Tests: answers with and without arranged imports agree on each backend, and the backends agree, for a join and a reduce of an import at the root, a join inside a loop, and the import read as rows, through changes and an importer installed late. A declared import of an export of unknown shape fails on mismatched data when read in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry
force-pushed
the
arranged-imports
branch
from
October 12, 2026 21:20
4c750e9 to
fc001f5
Compare
frankmcsherry
marked this pull request as ready for review
October 12, 2026 23:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A program that imported a published trace received it as rows (or, in columnar mode, as containers), and each of its joins, reduces, and arranges of the import built a private copy of an arrangement the producer already maintains. Imports now arrive as that arrangement, entered into the program's scope (
TraceEnter), on both backends.vec:
Arris a local or an imported arrangement. Join and reduce take either through the cursor tactics, as an entered batch's cursor lifts its times.corgi: times are integer lanes, and a host epoch
eand the program time(e, [])have the same lanes, so an imported chunk is read at the program's time by sharing it:CorgiChunk::retimecasts theRc, whichrepr(C)on the chunk's contents andColTimesmakes sound.RetimeJoinandRetimeReducewrap the proxy tactics and retime imported batches as they cross in, oneRcper chunk. Columnar exports, generated sources included, reach columnar programs this way.An imported arrangement also enters child scopes, iterative ones included, as a program's own arrangements do (#931). An import that a scope also reads as rows is flattened once per scope, and the rows enter its children with the arrangement.
Server::set_arranged_imports(false), or DDIR_ARRANGED_IMPORTS=0 for the server binary, restores rows (containers in columnar mode). A shape ascription on an arranged import is checked by a side operator over the trace's batches, which produces nothing: on vec always, on corgi when the export's shape was not known at install (as #927 checks containers).examples/import_bench: one producer of 1M edges, then 8 consumers each joining the import twice, 1000 edges replaced per tick. Corgi without arranged imports reads containers (#927).
Consumers that join the import inside a loop (reachability) are alike. After 500 ticks of churn the published trace holds the same updates with 0 or 8 importers sharing it.
Tests: answers with and without arranged imports agree on each backend, and the backends agree, for a join and a reduce of an import at the root, a join inside a loop, and the import read as rows, through changes and an importer installed late. A declared import of an export of unknown shape fails on mismatched data when read in place.