Skip to content

DDIR server: imports arrive as the producer's arrangement - #918

Merged
frankmcsherry merged 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:arranged-imports
Oct 12, 2026
Merged

frankmcsherry merged 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:arranged-imports

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 2, 2026 •

Copy link
Copy Markdown
Member

A program that imported a published trace received it as rows (or, in columnar mode, as containers), and each of its joins, reduces, and arranges of the import built a private copy of an arrangement the producer already maintains. Imports now arrive as that arrangement, entered into the program's scope (TraceEnter), on both backends.

vec: Arr is a local or an imported arrangement. Join and reduce take either through the cursor tactics, as an entered batch's cursor lifts its times.

corgi: times are integer lanes, and a host epoch e and the program time (e, []) have the same lanes, so an imported chunk is read at the program's time by sharing it: CorgiChunk::retime casts the Rc, which repr(C) on the chunk's contents and ColTimes makes sound. RetimeJoin and RetimeReduce wrap the proxy tactics and retime imported batches as they cross in, one Rc per chunk. Columnar exports, generated sources included, reach columnar programs this way.

An imported arrangement also enters child scopes, iterative ones included, as a program's own arrangements do (#931). An import that a scope also reads as rows is flattened once per scope, and the rows enter its children with the arrangement.

Server::set_arranged_imports(false), or DDIR_ARRANGED_IMPORTS=0 for the server binary, restores rows (containers in columnar mode). A shape ascription on an arranged import is checked by a side operator over the trace's batches, which produces nothing: on vec always, on corgi when the export's shape was not known at install (as #927 checks containers).

examples/import_bench: one producer of 1M edges, then 8 consumers each joining the import twice, 1000 edges replaced per tick. Corgi without arranged imports reads containers (#927).

                             vec              corgi
                          rows   arranged   containers   arranged
install-to-answer, ms     ~400   ~3         27-42        ~1
memory per consumer, MiB  ~95    ~0.5       ~54          ~0.3
tick, 8 consumers, ms     13.3   2.0        3.1          0.9

Consumers that join the import inside a loop (reachability) are alike. After 500 ticks of churn the published trace holds the same updates with 0 or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and the backends agree, for a join and a reduce of an import at the root, a join inside a loop, and the import read as rows, through changes and an importer installed late. A declared import of an export of unknown shape fails on mismatched data when read in place.

@frankmcsherry
frankmcsherry changed the base branch from master-next to ddir-share-arrangements October 12, 2026 20:36
@frankmcsherry
frankmcsherry changed the base branch from ddir-share-arrangements to master-next October 12, 2026 20:41
A program that imported a published trace received it as rows (or, in
columnar mode, as containers), and each of its joins, reduces, and arranges
of the import built a private copy of an arrangement the producer already
maintains. Imports now arrive as that arrangement, entered into the program's
scope (`TraceEnter`), on both backends.

vec: `Arr` is a local or an imported arrangement. Join and reduce take
either through the cursor tactics, as an entered batch's cursor lifts its
times.

corgi: times are integer lanes, and a host epoch `e` and the program time
`(e, [])` have the same lanes, so an imported chunk is read at the program's
time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on
the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and
`RetimeReduce` wrap the proxy tactics and retime imported batches as they
cross in, one `Rc` per chunk. Columnar exports, generated sources included,
reach columnar programs this way.

An imported arrangement also enters child scopes, iterative ones included,
as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads
as rows is flattened once per scope, and the rows enter its children with the
arrangement.

`Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the
server binary, restores rows (containers in columnar mode). A shape
ascription on an arranged import is checked by a side operator over the
trace's batches, which produces nothing: on vec always, on corgi when the
export's shape was not known at install (as TimelyDataflow#927 checks containers).

examples/import_bench: one producer of 1M edges, then 8 consumers each
joining the import twice, 1000 edges replaced per tick. Corgi without
arranged imports reads containers (TimelyDataflow#927).

                             vec              corgi
                          rows   arranged   containers   arranged
install-to-answer, ms     ~400   ~3         27-42        ~1
memory per consumer, MiB  ~95    ~0.5       ~54          ~0.3
tick, 8 consumers, ms     13.3   2.0        3.1          0.9

Consumers that join the import inside a loop (reachability) are alike.
After 500 ticks of churn the published trace holds the same updates with 0
or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and
the backends agree, for a join and a reduce of an import at the root, a join
inside a loop, and the import read as rows, through changes and an importer
installed late. A declared import of an export of unknown shape fails on
mismatched data when read in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@frankmcsherry
frankmcsherry marked this pull request as ready for review October 12, 2026 23:32
@frankmcsherry
frankmcsherry merged commit cece909 into TimelyDataflow:master-next Oct 12, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant