Skip to content

feat: apply write defaults in DataWriter - #942

Merged
wgtmac merged 14 commits into
apache:mainfrom
huan233usc:feat/write-default-e2e
Oct 30, 2026
Merged

wgtmac merged 14 commits into
apache:mainfrom
huan233usc:feat/write-default-e2e

Conversation

@huan233usc

@huan233usc huan233usc commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Writing a batch with an older schema should fill omitted columns with Iceberg
write-default values. This PR adds optional DataWriterOptions::input_schema;
for a required column with initial-default 42 and write-default 7, input
[[1], [2]] writes [[1, 7], [2, 7]].

DataWriter prepares a reusable WriteProjection once in Make and applies it
to each batch. The projection matches fields by ID recursively, preserves
supplied values and explicit nulls, fills missing optional fields with typed
nulls, and materializes missing fields from their write-default. When no
input_schema is provided, or the schemas are identical, the original Arrow
batch is passed through without projection.

Incompatible schemas fail before opening the file, including extra input IDs,
missing required fields without a write-default, optional-to-required fields,
and concrete-to-unknown writes. Existing primitive types must match; read
projection behavior is unchanged.

Parquet and Avro writers remain unchanged and continue to expect batches
matching their complete target schema. The shared projection is implemented in
core using nanoarrow shared buffers without adding an Arrow C++ dependency.

Addresses the write-path portion of #637.

Validation: data_test (184 passed), schema_test (576 passed), avro_test
(238 passed), and parquet_test (110 passed, 1 existing skip). Tests cover two
consecutive batches per format, nested slices and parent nulls, primitive
defaults, explicit null preservation, independent child ownership, and schema
rejection at DataWriter::Make. A separate core/data shared-library build with
bundle disabled passed. clang-format, clang-tidy, git diff --check, and the
Windows CI matrix passed.

@huan233usc
huan233usc force-pushed the feat/write-default-e2e branch 7 times, most recently from b824448 to ddb87cc Compare September 18, 2026 22:45
Extend Project with ProjectionOptions so the same field-id matching
serves both paths: reads fill missing fields with initial-default and
allow type promotion, writes fill with write-default and require exact
types.
Add arrow::AlignBatchForWrite which projects an input batch onto the
write schema via Project with write-default options: present columns
are reordered by field id (recursively for structs), missing columns
are materialized from write-default or null, and unknown input columns
are rejected rather than silently dropped.
DataWriter aligns batches to the write schema when the input schema
differs, via a new Write(data, input_schema) overload; schemas that
already match keep the zero-conversion Write(data) path. Adds Parquet
and Avro end-to-end tests where a column added with write-default
reads back the default for rows written without it.
@wgtmac

wgtmac commented Oct 9, 2026

Copy link
Copy Markdown
Member

Is this ready for review?

@huan233usc

Copy link
Copy Markdown
Contributor Author

I plan to do some final passes and then will send for review -- thanks

- Preserve sliced struct offsets and align collection children by field id.
- Materialize unknown inputs as typed nulls.
- Configure optional input schemas without extending the writer interface.
- Always align batches while retaining the declared or default input schema.
- Forward equal-schema batches without importing or exporting Arrow C Data.
@huan233usc
huan233usc marked this pull request as ready for review October 12, 2026 16:48
Comment thread src/iceberg/schema_util.cc
Comment thread src/iceberg/avro/avro_writer.cc Outdated
Reject optional inputs for required write fields, including nested fields. Create writer guards after alignment and verify rejection paths release input exactly once.
Comment thread src/iceberg/arrow/arrow_c_data_util.cc Outdated
Comment thread src/iceberg/arrow_c_data_util_internal.h Outdated

@wgtmac wgtmac left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After taking a closer look, I'd keep schema alignment and default filling outside the Parquet/Avro writers. They should continue to expect batches matching the full write schema and write the supplied values as-is. Callers can materialize write-default values before invoking the format writer, or DataWriter can handle this through a shared adapter that prepares the projection once and applies it to each batch.

Keep schema alignment in DataWriter and restore format writer boundaries.
Use core nanoarrow shared buffers and validate write schemas at creation.
Reject concrete-to-unknown writes and consolidate write coverage.
@huan233usc huan233usc changed the title feat: apply write defaults in Parquet and Avro writers feat: apply write defaults in DataWriter Oct 28, 2026
@wgtmac

wgtmac commented Oct 30, 2026

Copy link
Copy Markdown
Member

Huge thanks to @huan233usc for adding this and completing the v3 default value support! Thanks @shangxinli for the review!

@wgtmac
wgtmac merged commit 942e9af into apache:main Oct 30, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants