Skip to content

DDIR: child scopes read their parent's arrangements - #931

Merged
frankmcsherry merged 1 commit into
master-nextfrom
ddir-share-arrangements
Oct 12, 2026
Merged

DDIR: child scopes read their parent's arrangements#931
frankmcsherry merged 1 commit into
master-nextfrom
ddir-share-arrangements

Conversation

@frankmcsherry

@frankmcsherry frankmcsherry commented Oct 12, 2026

Copy link
Copy Markdown
Member

A child scope used to receive every import as rows.
An arrangement in the parent was flattened at the scope boundary, and each join, reduce, or arrange of it inside the child built a new one.
Imports now arrive as the parent rendered them, and an arrangement enters the child's region with Arranged::enter_region, sharing the trace.

Why no TraceEnter

DD's TraceEnter maps a reader's compaction request back to the outer time because the trace cannot hold inner times.
DDIR uses one time type, Product<u64, PointStamp<u64>>, in every scope, so an inner request is already a valid frontier for the outer trace.
It is also sound: each handle's frontier lower bounds the times its reader will ask about, and the trace compacts to the meet of its handles.
While epoch e iterates at round i and epoch e+1 may still arrive, a loop reader's frontier is {(e,[i]), (e+1,[])}, and advancing an outer time (e',[]) with e' < e by it gives (e,[]).

Changes

  • Backend::enter_region, one line in each backend.
  • render_tree renders a child's imports as the parent's items (collection or arrangement) rather than rows; its public signature is unchanged.

Sharing happens when the parent item is an arrangement: an explicit arrange, or a reduce.
lower places the arrange for a join in the joining scope, so a child joining a parent's collection still arranges it unless the parent does.
Among the checked-in programs, AoC 2023 day 4 part 2 (matches) and day 22 part 2 (supcnt) each now share a reduce's output with a child instead of rebuilding it.

This is the in-program half of #918, which can rebase onto it and drop its shares_into_regions gate.

Tests

  • tests/shared_arrangements.rs: reach and SCC, as written and with the parent arranging the collections its children join, through 8 epochs of edge insertions and deletions; the shared variants agree with the written one at every epoch on vec and corgi, at 1 and 3 workers.
  • cargo test -p interactive and cargo test -p ddir-server pass; AoC 2023 run.sh passes on both backends.

🤖 Generated with Claude Code

@frankmcsherry @claude
DDIR: child scopes read their parent's arrangements
f09a877 
A child scope used to receive every import as rows.
An arrangement in the parent was flattened at the boundary, and each join, reduce, or arrange of it inside the child built a new one.
Imports now arrive as the parent rendered them, and an arrangement enters the child's region with `Arranged::enter_region`, sharing the trace.

Every scope has the same time type, so no `TraceEnter` is needed, and compaction requests need no translation.
A reader inside a loop asks for compaction at iteration times, which is sound: each handle's frontier lower bounds the times its reader will ask about, and the trace compacts to the meet of its handles.

Sharing happens when the parent item is an arrangement: an explicit `arrange`, or a reduce.
`lower` places the arrange for a join in the joining scope, so a child joining a parent's collection still arranges it unless the parent does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry added a commit to frankmcsherry/differential-dataflow that referenced this pull request Oct 12, 2026
@frankmcsherry @claude
DDIR server: imports arrive as the producer's arrangement
a0d249a 
A program that imported a published trace received it as rows, and each of
its joins, reduces, and arranges of the import built a private copy of an
arrangement the producer already maintains. Imports now arrive as that
arrangement, entered into the program's scope (`TraceEnter`), on both
backends.

vec: `Arr` is a local or an imported arrangement. Join and reduce take
either through the cursor tactics, as an entered batch's cursor lifts its
times.

corgi: times are integer lanes, and a host epoch `e` and the program time
`(e, [])` have the same lanes, so an imported chunk is read at the program's
time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on
the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and
`RetimeReduce` wrap the proxy tactics and retime imported batches as they
cross in, one `Rc` per chunk. Columnar exports reach columnar programs this
way.

An imported arrangement also enters child scopes, iterative ones included,
as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads
as rows is flattened once per scope, and the rows enter its children with the
arrangement.

`Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the
server binary, restores rows. On vec a shape ascription is checked by a side
operator over the trace's batches; on corgi an arranged import's ascription
is not checked (its chunks carry the producer's shapes).

examples/import_bench: one producer of 1M edges, then 8 consumers each
joining the import twice, 1000 edges replaced per tick.

                             vec              corgi
                          rows   arranged   rows   arranged
install-to-answer, ms     ~420   ~3         ~100   ~1
memory per consumer, MiB  ~120   ~1         ~32    ~0.4
tick, 8 consumers, ms     13-19  1.6        3.9-4.7  0.8

Consumers that join the import inside a loop (reachability) are alike.
After 500 ticks of churn the published trace holds the same updates with 0
or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and
the backends agree, for a join and a reduce of an import at the root, a join
inside a loop, and the import read as rows, through changes and an importer
installed late.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@frankmcsherry frankmcsherry mentioned this pull request Oct 12, 2026
Merged
@frankmcsherry
frankmcsherry merged commit e768bef into master-next Oct 12, 2026
6 checks passed
frankmcsherry added a commit to frankmcsherry/differential-dataflow that referenced this pull request Oct 12, 2026
@frankmcsherry @claude
DDIR server: imports arrive as the producer's arrangement
4c750e9 
A program that imported a published trace received it as rows, and each of
its joins, reduces, and arranges of the import built a private copy of an
arrangement the producer already maintains. Imports now arrive as that
arrangement, entered into the program's scope (`TraceEnter`), on both
backends.

vec: `Arr` is a local or an imported arrangement. Join and reduce take
either through the cursor tactics, as an entered batch's cursor lifts its
times.

corgi: times are integer lanes, and a host epoch `e` and the program time
`(e, [])` have the same lanes, so an imported chunk is read at the program's
time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on
the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and
`RetimeReduce` wrap the proxy tactics and retime imported batches as they
cross in, one `Rc` per chunk. Columnar exports reach columnar programs this
way.

An imported arrangement also enters child scopes, iterative ones included,
as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads
as rows is flattened once per scope, and the rows enter its children with the
arrangement.

`Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the
server binary, restores rows. On vec a shape ascription is checked by a side
operator over the trace's batches; on corgi an arranged import's ascription
is not checked (its chunks carry the producer's shapes).

examples/import_bench: one producer of 1M edges, then 8 consumers each
joining the import twice, 1000 edges replaced per tick.

                             vec              corgi
                          rows   arranged   rows   arranged
install-to-answer, ms     ~420   ~3         ~100   ~1
memory per consumer, MiB  ~120   ~1         ~32    ~0.4
tick, 8 consumers, ms     13-19  1.6        3.9-4.7  0.8

Consumers that join the import inside a loop (reachability) are alike.
After 500 ticks of churn the published trace holds the same updates with 0
or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and
the backends agree, for a join and a reduce of an import at the root, a join
inside a loop, and the import read as rows, through changes and an importer
installed late.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
frankmcsherry added a commit to frankmcsherry/differential-dataflow that referenced this pull request Oct 12, 2026
@frankmcsherry @claude
DDIR server: imports arrive as the producer's arrangement
fc001f5 
A program that imported a published trace received it as rows (or, in
columnar mode, as containers), and each of its joins, reduces, and arranges
of the import built a private copy of an arrangement the producer already
maintains. Imports now arrive as that arrangement, entered into the program's
scope (`TraceEnter`), on both backends.

vec: `Arr` is a local or an imported arrangement. Join and reduce take
either through the cursor tactics, as an entered batch's cursor lifts its
times.

corgi: times are integer lanes, and a host epoch `e` and the program time
`(e, [])` have the same lanes, so an imported chunk is read at the program's
time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on
the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and
`RetimeReduce` wrap the proxy tactics and retime imported batches as they
cross in, one `Rc` per chunk. Columnar exports, generated sources included,
reach columnar programs this way.

An imported arrangement also enters child scopes, iterative ones included,
as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads
as rows is flattened once per scope, and the rows enter its children with the
arrangement.

`Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the
server binary, restores rows (containers in columnar mode). A shape
ascription on an arranged import is checked by a side operator over the
trace's batches, which produces nothing: on vec always, on corgi when the
export's shape was not known at install (as TimelyDataflow#927 checks containers).

examples/import_bench: one producer of 1M edges, then 8 consumers each
joining the import twice, 1000 edges replaced per tick. Corgi without
arranged imports reads containers (TimelyDataflow#927).

                             vec              corgi
                          rows   arranged   containers   arranged
install-to-answer, ms     ~400   ~3         27-42        ~1
memory per consumer, MiB  ~95    ~0.5       ~54          ~0.3
tick, 8 consumers, ms     13.3   2.0        3.1          0.9

Consumers that join the import inside a loop (reachability) are alike.
After 500 ticks of churn the published trace holds the same updates with 0
or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and
the backends agree, for a join and a reduce of an import at the root, a join
inside a loop, and the import read as rows, through changes and an importer
installed late. A declared import of an export of unknown shape fails on
mismatched data when read in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry added a commit that referenced this pull request Oct 12, 2026
@frankmcsherry @claude
DDIR server: imports arrive as the producer's arrangement (#918)
cece909 
A program that imported a published trace received it as rows (or, in
columnar mode, as containers), and each of its joins, reduces, and arranges
of the import built a private copy of an arrangement the producer already
maintains. Imports now arrive as that arrangement, entered into the program's
scope (`TraceEnter`), on both backends.

vec: `Arr` is a local or an imported arrangement. Join and reduce take
either through the cursor tactics, as an entered batch's cursor lifts its
times.

corgi: times are integer lanes, and a host epoch `e` and the program time
`(e, [])` have the same lanes, so an imported chunk is read at the program's
time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on
the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and
`RetimeReduce` wrap the proxy tactics and retime imported batches as they
cross in, one `Rc` per chunk. Columnar exports, generated sources included,
reach columnar programs this way.

An imported arrangement also enters child scopes, iterative ones included,
as a program's own arrangements do (#931). An import that a scope also reads
as rows is flattened once per scope, and the rows enter its children with the
arrangement.

`Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the
server binary, restores rows (containers in columnar mode). A shape
ascription on an arranged import is checked by a side operator over the
trace's batches, which produces nothing: on vec always, on corgi when the
export's shape was not known at install (as #927 checks containers).

examples/import_bench: one producer of 1M edges, then 8 consumers each
joining the import twice, 1000 edges replaced per tick. Corgi without
arranged imports reads containers (#927).

                             vec              corgi
                          rows   arranged   containers   arranged
install-to-answer, ms     ~400   ~3         27-42        ~1
memory per consumer, MiB  ~95    ~0.5       ~54          ~0.3
tick, 8 consumers, ms     13.3   2.0        3.1          0.9

Consumers that join the import inside a loop (reachability) are alike.
After 500 ticks of churn the published trace holds the same updates with 0
or 8 importers sharing it.

Tests: answers with and without arranged imports agree on each backend, and
the backends agree, for a join and a reduce of an import at the root, a join
inside a loop, and the import read as rows, through changes and an importer
installed late. A declared import of an export of unknown shape fails on
mismatched data when read in place.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@frankmcsherry