Repository navigation
feat: apply write defaults in DataWriter - #942
Merged
Merged
Conversation
huan233usc
force-pushed
the
feat/write-default-e2e
branch
7 times, most recently
from
September 18, 2026 22:45
b824448 to
ddb87cc
Compare
Extend Project with ProjectionOptions so the same field-id matching serves both paths: reads fill missing fields with initial-default and allow type promotion, writes fill with write-default and require exact types.
Add arrow::AlignBatchForWrite which projects an input batch onto the write schema via Project with write-default options: present columns are reordered by field id (recursively for structs), missing columns are materialized from write-default or null, and unknown input columns are rejected rather than silently dropped.
DataWriter aligns batches to the write schema when the input schema differs, via a new Write(data, input_schema) overload; schemas that already match keep the zero-conversion Write(data) path. Adds Parquet and Avro end-to-end tests where a column added with write-default reads back the default for rows written without it.
huan233usc
force-pushed
the
feat/write-default-e2e
branch
from
September 18, 2026 22:57
ddb87cc to
8e01c0e
Compare
14 of 21 tasks
8 tasks
Member
|
Is this ready for review? |
Contributor
Author
|
I plan to do some final passes and then will send for review -- thanks |
- Preserve sliced struct offsets and align collection children by field id. - Materialize unknown inputs as typed nulls. - Configure optional input schemas without extending the writer interface.
- Always align batches while retaining the declared or default input schema. - Forward equal-schema batches without importing or exporting Arrow C Data.
huan233usc
marked this pull request as ready for review
October 12, 2026 16:48
shangxinli
reviewed
Oct 17, 2026
wgtmac
reviewed
Oct 18, 2026
Reject optional inputs for required write fields, including nested fields. Create writer guards after alignment and verify rejection paths release input exactly once.
wgtmac
reviewed
Oct 20, 2026
wgtmac
reviewed
Oct 20, 2026
wgtmac
requested changes
Oct 28, 2026
wgtmac
left a comment
Member
There was a problem hiding this comment.
After taking a closer look, I'd keep schema alignment and default filling outside the Parquet/Avro writers. They should continue to expect batches matching the full write schema and write the supplied values as-is. Callers can materialize write-default values before invoking the format writer, or DataWriter can handle this through a shared adapter that prepares the projection once and applies it to each batch.
Keep schema alignment in DataWriter and restore format writer boundaries. Use core nanoarrow shared buffers and validate write schemas at creation. Reject concrete-to-unknown writes and consolidate write coverage.
wgtmac
approved these changes
Oct 30, 2026
Member
|
Huge thanks to @huan233usc for adding this and completing the v3 default value support! Thanks @shangxinli for the review! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Writing a batch with an older schema should fill omitted columns with Iceberg
write-default values. This PR adds optional
DataWriterOptions::input_schema;for a required column with initial-default 42 and write-default 7, input
[[1], [2]]writes[[1, 7], [2, 7]].DataWriter prepares a reusable
WriteProjectiononce inMakeand applies itto each batch. The projection matches fields by ID recursively, preserves
supplied values and explicit nulls, fills missing optional fields with typed
nulls, and materializes missing fields from their write-default. When no
input_schemais provided, or the schemas are identical, the original Arrowbatch is passed through without projection.
Incompatible schemas fail before opening the file, including extra input IDs,
missing required fields without a write-default, optional-to-required fields,
and concrete-to-unknown writes. Existing primitive types must match; read
projection behavior is unchanged.
Parquet and Avro writers remain unchanged and continue to expect batches
matching their complete target schema. The shared projection is implemented in
core using nanoarrow shared buffers without adding an Arrow C++ dependency.
Addresses the write-path portion of #637.
Validation: data_test (184 passed), schema_test (576 passed), avro_test
(238 passed), and parquet_test (110 passed, 1 existing skip). Tests cover two
consecutive batches per format, nested slices and parent nulls, primitive
defaults, explicit null preservation, independent child ownership, and schema
rejection at
DataWriter::Make. A separate core/data shared-library build withbundle disabled passed. clang-format, clang-tidy, git diff --check, and the
Windows CI matrix passed.