[YinZheng-Sun]

Identity-partitioned source columns may be omitted from data files because their values are stored in the partition tuple. This PR carries each file’s writing spec through scan planning, JSON/REST tasks and splits, and supplies identity source constants to Parquet and Avro readers.

Reading behavior

• Build a per-file id_to_constant map keyed by source field ID, using the task’s own historical PartitionSpec. Keep the spec pointer consistent with the other task fields.
• Pass constants through the existing Project entry points and select them during recursive field projection, including primitive leaves of existing structs. There is no separate ProjectConstant or ValidateConstant method. Like the Java reader path, projection trusts the supplied constants.
• Give constants precedence over physical values and initial defaults, preserving explicit nulls. Materialize constants before equality-delete evaluation, including columns needed by deletes but omitted from the output.
• Preserve upstream row-group filtering, filter binding and case sensitivity. Keep the original two-argument metrics filter interface unchanged; partition constants participate only in projection.

Scope

Identity source columns only: bucket and other transforms do not reconstruct source values. _partition, _spec_id and _file are not added to this constant map. Nested/struct constant values and reconstruction of an entirely absent parent struct are outside this change. JSON tasks must retain the historical source-type information needed to decode partition values.

The reader flow follows Java’s PartitionUtil.constantsMap, GenericReader and recursive Parquet reader construction. Feedback on spec ownership and the constant-map interface is welcome.

Validation

Merged upstream main at 748162b3 and resolved the task-reader and Parquet-reader conflicts. After restoring the original metrics filter interface, rebuilt and passed data_test (200) and parquet_test (193), plus pre-commit and clang-tidy 22.1.8 on the changed reader and metrics filter. Earlier validation also passed schema_test (566) and avro_test (255).

For local Apple-toolchain testing only, an existing upstream std::ranges::iota call was temporarily replaced with equivalent std::iota; the source was restored and that workaround is not committed. Clang-tidy 22.1.8 found two diagnostics in the external Avro GenericDatum.hh; both were reproduced against unmodified upstream avro_data_util.cc.

[advancedxy]

Identity-partitioned source columns may be omitted from data files because their values are stored in the partition tuple.

for the record: apache/iceberg#10835 this might be the most relevant discussion and per iceberg spec: partition source data columns might be always produced unless the data files are migrated from hive tables.

However java's reference implementation prefers id constant from partition data.