Repository navigation
DDIR server: keep inputs, generated sources, imports, and binds as corgi columns - #927
Merged
Merged
Conversation
…rgi columns With the corgi backend and columnar exports, a program's data no longer passes through rows on its way in. Inputs convert fed rows to columns a batch at a time, at their declared, recipe, or first-row shape. Generated sources and the clock build their columns directly and publish as corgi chunk traces. Imports read chunk traces as containers and enter the program scope without a row conversion. A bind from a columnar trace into a column input carries containers. Only snapshots form rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/differential-dataflow
that referenced
this pull request
Oct 12, 2026
A program that imported a published trace received it as rows (or, in columnar mode, as containers), and each of its joins, reduces, and arranges of the import built a private copy of an arrangement the producer already maintains. Imports now arrive as that arrangement, entered into the program's scope (`TraceEnter`), on both backends. vec: `Arr` is a local or an imported arrangement. Join and reduce take either through the cursor tactics, as an entered batch's cursor lifts its times. corgi: times are integer lanes, and a host epoch `e` and the program time `(e, [])` have the same lanes, so an imported chunk is read at the program's time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and `RetimeReduce` wrap the proxy tactics and retime imported batches as they cross in, one `Rc` per chunk. Columnar exports, generated sources included, reach columnar programs this way. An imported arrangement also enters child scopes, iterative ones included, as a program's own arrangements do (TimelyDataflow#931). An import that a scope also reads as rows is flattened once per scope, and the rows enter its children with the arrangement. `Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the server binary, restores rows (containers in columnar mode). A shape ascription on an arranged import is checked by a side operator over the trace's batches, which produces nothing: on vec always, on corgi when the export's shape was not known at install (as TimelyDataflow#927 checks containers). examples/import_bench: one producer of 1M edges, then 8 consumers each joining the import twice, 1000 edges replaced per tick. Corgi without arranged imports reads containers (TimelyDataflow#927). vec corgi rows arranged containers arranged install-to-answer, ms ~400 ~3 27-42 ~1 memory per consumer, MiB ~95 ~0.5 ~54 ~0.3 tick, 8 consumers, ms 13.3 2.0 3.1 0.9 Consumers that join the import inside a loop (reachability) are alike. After 500 ticks of churn the published trace holds the same updates with 0 or 8 importers sharing it. Tests: answers with and without arranged imports agree on each backend, and the backends agree, for a join and a reduce of an import at the root, a join inside a loop, and the import read as rows, through changes and an importer installed late. A declared import of an export of unknown shape fails on mismatched data when read in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frankmcsherry
added a commit
that referenced
this pull request
Oct 12, 2026
A program that imported a published trace received it as rows (or, in columnar mode, as containers), and each of its joins, reduces, and arranges of the import built a private copy of an arrangement the producer already maintains. Imports now arrive as that arrangement, entered into the program's scope (`TraceEnter`), on both backends. vec: `Arr` is a local or an imported arrangement. Join and reduce take either through the cursor tactics, as an entered batch's cursor lifts its times. corgi: times are integer lanes, and a host epoch `e` and the program time `(e, [])` have the same lanes, so an imported chunk is read at the program's time by sharing it: `CorgiChunk::retime` casts the `Rc`, which `repr(C)` on the chunk's contents and `ColTimes` makes sound. `RetimeJoin` and `RetimeReduce` wrap the proxy tactics and retime imported batches as they cross in, one `Rc` per chunk. Columnar exports, generated sources included, reach columnar programs this way. An imported arrangement also enters child scopes, iterative ones included, as a program's own arrangements do (#931). An import that a scope also reads as rows is flattened once per scope, and the rows enter its children with the arrangement. `Server::set_arranged_imports(false)`, or DDIR_ARRANGED_IMPORTS=0 for the server binary, restores rows (containers in columnar mode). A shape ascription on an arranged import is checked by a side operator over the trace's batches, which produces nothing: on vec always, on corgi when the export's shape was not known at install (as #927 checks containers). examples/import_bench: one producer of 1M edges, then 8 consumers each joining the import twice, 1000 edges replaced per tick. Corgi without arranged imports reads containers (#927). vec corgi rows arranged containers arranged install-to-answer, ms ~400 ~3 27-42 ~1 memory per consumer, MiB ~95 ~0.5 ~54 ~0.3 tick, 8 consumers, ms 13.3 2.0 3.1 0.9 Consumers that join the import inside a loop (reachability) are alike. After 500 ticks of churn the published trace holds the same updates with 0 or 8 importers sharing it. Tests: answers with and without arranged imports agree on each backend, and the backends agree, for a join and a reduce of an import at the root, a join inside a loop, and the import read as rows, through changes and an importer installed late. A declared import of an export of unknown shape fails on mismatched data when read in place. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With the corgi backend and columnar exports (the default), a program's data no longer passes through rows on its way in.
Profiles of programs over generated sources put 33-56% of worker time in a row
MergeBatcherarranging those sources, before the corgi backend converted the rows to columns again.Changes
ServerInputis a row session or aColumnInput.A
ColumnInputbuffers fed rows and converts them to columns 64K at a time, at the input's declared shape, a recipe's shape, or the first row's.Recipe::containerbuilds a generated source's columns directly, never forming rows (gen_field_seededis split out ofgen_row_seeded).Generated sources and
clockshareinstall_sourceand publish corgi chunk traces througharrange_export.Recipe loads into program inputs and per-tick churn produce columns too.
A columnar install reads chunk traces as containers (
export_containers) and enters the program scope through a newEnter<u64, Time>(ColTimes::retime).A declared import of a trace whose shape is unknown at install is checked per container (
assert_shape), as the row path checked each row.A bind from a columnar trace into a column input captures containers, and
tickgives them to the input at the new epoch.Other source/target pairs keep rows.
In columnar mode only
snapshotforms rows.trace_infonow lists columnar traces, whichlistwas missing.The row path is unchanged when columnar exports are off or the backend is vec.
Measurements
Single worker, release, codegen-units=1, whole process, three interleaved runs, identical outputs:
random:nodes=1000000,edges=4000000random:nodes=1000000,edges=2000000random:nodes=400000,edges=4000000Programs fed through
input(e.g.scc.ddpwithfeed p 0 from random:...) never arranged rows; for them only the row generation and conversion go away.Tests
tests/server_columnar_sources.rsruns churningrandom:,iota:,clock, a churning recipe load, a declared import of an undeclared export, and a bind, at 1 and 3 workers, and compares the vec backend, corgi over rows, and corgi over columns.A unit test checks that a recipe's columns are its rows.
🤖 Generated with Claude Code