feat(inspect): support selecting columns in metadata table scans by manuzhang · Pull Request #1008 · apache/iceberg-cpp

manuzhang · 2026-10-13T15:42:47Z

What

Add ColumnSelection (column names plus case sensitivity), MetadataTable::ScanColumns(const ColumnSelection&), and TimeTravelMetadataTable::ScanColumns(const SnapshotSelection&, const ColumnSelection&). Names resolve through Schema::Select, as in TableScanBuilder::Select: nested names are dot-concatenated, columns come back in schema order, * selects all columns, and unknown names are ignored (as in Java's Schema.select). An empty list returns rows without columns, which is enough to count rows. Selecting a map's value returns the whole map; selecting only its key is an error.
Scan() is no longer virtual. Tables implement the protected ScanProjection(const Schema& projected_schema) instead, and ScanSnapshot also receives the projected schema.
Add inspect/metadata_table_columns_internal.h, which maps the projected columns to per-column append functions so a table builds only the selected columns. Selecting only some fields of a nested struct column returns NotSupported for now; no current metadata table has a struct column.
SnapshotsTable builds only the selected columns. HistoryTable scans still return NotSupported.

Why

Metadata table scans currently always return every column. Java's metadata tables support newScan().select(...): StaticDataTask projects each row to scan.schema(), so an engine running SELECT name FROM t.refs only gets name. Without selection, a C++ caller that needs one column still pays to build all of them, for example copying every snapshot's summary map.

Behavior change

New ScanColumns methods return only the selected columns. Scan() and Scan(const SnapshotSelection&) return the same rows as before, and Scan({}) on a time-travel table still resolves to Scan(const SnapshotSelection&).
Metadata table implementations override ScanProjection(const Schema&) (or ScanSnapshot(const SnapshotSelection&, const Schema&) for time-travel tables) instead of Scan(). Open metadata table PRs (feat(inspect): add files, partitions, and manifests metadata tables #876, feat: add position_deletes metadata table #920, feat(inspect): add refs metadata table #1007) need the same change.

Testing

Added metadata_table_test cases for SnapshotsTable covering selection order, unknown names, case sensitivity, * and no selection, an empty selection, and map columns (summary.value returns the whole map, summary.key is an error). A recording TimeTravelMetadataTable checks the snapshot selection and projected columns passed by Scan(), ScanColumns, Scan(const SnapshotSelection&), and Scan({}), and another case checks that HistoryTable::ScanColumns returns NotSupported.

🤖 Generated with Claude Code

Add ColumnSelection and Scan overloads that return only the requested columns, matching Java's metadata table select(). Names resolve through Schema::Select, so nested names, case sensitivity, `*`, and unknown names behave as in TableScanBuilder::Select. Tables now implement ScanProjection(const Schema&) and build only the selected columns. SnapshotsTable uses a per-column append table from the new metadata_table_columns_internal.h helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

ICEBERG_CHECK is used for conditions that can be triggered by input/projection mismatch (e.g., a… · New
This assertion hard-codes the expected map size (10), which makes the test brittle if the fixture… · New

What changed in this PR

Adds column projection support to C++ Iceberg metadata table scans (including time travel), aligning behavior with Java (newScan().select(...)) and enabling engines to request only needed columns.

Changes:

Introduces ColumnSelection and overloads of MetadataTable::Scan(...) / TimeTravelMetadataTable::Scan(...) to project columns via Schema::Select.
Refactors scanning to be implemented via ScanProjection(const Schema&) / ScanSnapshot(..., const Schema&) instead of overriding Scan().
Adds internal helpers to map projected schema columns to per-column append functions, and updates SnapshotsTable + tests accordingly.

File	Description
src/iceberg/inspect/metadata_table.h	Adds `ColumnSelection` and new `Scan` overloads; refactors scan implementation to `ScanProjection` / projected-schema time travel.
src/iceberg/inspect/metadata_table.cc	Implements column selection via `Schema::Select` and routes scans through projected schemas.
src/iceberg/inspect/metadata_table_columns_internal.h	New internal helper to select per-column append functions based on projected schema.
src/iceberg/inspect/snapshots_table.h	Updates `SnapshotsTable` to implement `ScanProjection(...)` rather than `Scan()`.
src/iceberg/inspect/snapshots_table.cc	Implements projected-column building for snapshots using selected append functions.
src/iceberg/inspect/history_table.h / .cc	Updates `HistoryTable` to `ScanProjection(...)` while keeping scan unsupported.
src/iceberg/test/snapshots_table_test.cc	Adds tests for column selection semantics on `SnapshotsTable`.
src/iceberg/test/metadata_table_test.cc	Adds tests for time-travel scans with column projection and `HistoryTable` behavior.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    ICEBERG_CHECK(field.has_value(), "Metadata table column '{}' is not in the schema",
+                  projected_field.name());
+    if (*field->get().type() != *projected_field.type()) {
+      return NotSupported("Cannot select nested fields of metadata table column '{}'",
+                          projected_field.name());
+    }
+    auto column = std::ranges::find(columns, projected_field.field_id(),
+                                    &MetadataTableColumn<Row>::field_id);
+    ICEBERG_CHECK(column != columns.end(),
+                  "Metadata table column '{}' has no append function",
+                  projected_field.name());


Scan(const ColumnSelection&) made Scan({}) on a TimeTravelMetadataTable ambiguous with Scan(const SnapshotSelection&), since both are aggregates. Use a distinct name so existing Scan({}) calls keep resolving to the snapshot-selection overload. Also assert on summary entries instead of the summary map's size, and move the selection rules explained in test comments to the ColumnSelection docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

shangxinli

The ScanColumns rename resolves the ambiguity I raised - Scan({}) still picks the SnapshotSelection overload. LGTM.

…ilder Replace ColumnSelection and the ScanColumns methods with a scan builder shaped like TableScanBuilder, as requested in review. MetadataTable::NewScan() returns a MetadataScanBuilder with Select(), CaseSensitive(), UseSnapshot(), UseRef(), and AsOfTime(). Build() validates the options, projects the schema, and returns a MetadataScan whose ToArrowArrayStream() reads the rows. Tables without time travel reject snapshot selections when the scan is built. Return InvalidArgument instead of ICEBERG_CHECK failures when a projected column has no append function, and look up projected columns with GetFieldById so nested field ids are rejected. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Make the scan surface a subset of the API proposed in apache#1066 so the two can converge: - MetadataTable::NewScan() is virtual and const, and ScanProjection() is const. - MetadataScanBuilder::Select() resolves names immediately with the current CaseSensitive() setting and records errors for Build(); Project() sets the output schema directly, and Build() rejects a null projection. - MetadataScan::ToArrowArrayStream() is virtual and const. - Projection validation rejects unknown columns, type changes, and projecting a nullable column as required, with the same messages as apache#1066. - Drop the time-travel options together with TimeTravelMetadataTable and SnapshotSelection, which apache#1066 also removes; nothing on main implements them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…Build() Drop the MetadataScan class: separating a built scan from the Arrow stream it reads belongs to the task-based design in apache#1066. Build() now validates the options and returns the stream directly, and each call returns a new stream. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Restore SnapshotSelection, TimeTravelMetadataTable, and supports_time_travel() and pass the snapshot selection through the builder as UseSnapshot(), UseRef(), and AsOfTime(), like TableScanBuilder. Build() returns NotSupported for tables that do not support time travel, and conflicting options are validation errors. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Copilot AI balanced review requested due to automatic review settings October 13, 2026 15:42

Copilot AI reviewed Oct 13, 2026

View reviewed changes

Copilot stopped reviewing on behalf of manuzhang due to an error October 13, 2026 20:17
Workflow completed but conclusion is not available. This may indicate rate limiting or API issues.

shangxinli reviewed Oct 14, 2026

View reviewed changes

Comment thread src/iceberg/inspect/metadata_table.h Outdated

Copilot AI balanced review requested due to automatic review settings October 15, 2026 11:30

Copilot AI reviewed Oct 15, 2026

shangxinli approved these changes Oct 15, 2026

View reviewed changes

manuzhang requested a review from wgtmac October 15, 2026 14:51

wgtmac reviewed Oct 16, 2026

View reviewed changes

Comment thread src/iceberg/inspect/metadata_table.h Outdated

manuzhang and others added 3 commits October 28, 2026 22:30

Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:12

Copilot AI reviewed Oct 28, 2026

Copilot AI balanced review requested due to automatic review settings October 28, 2026 15:32

Copilot AI reviewed Oct 28, 2026

manuzhang requested review from shangxinli and wgtmac October 28, 2026 15:36

Navigation Menu

Repository navigation

feat(inspect): support selecting columns in metadata table scans - #1008

feat(inspect): support selecting columns in metadata table scans#1008
manuzhang wants to merge 6 commits into
apache:mainfrom
manuzhang:metadata-table-select-columns

manuzhang commented Oct 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

shangxinli left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

manuzhang commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Behavior change

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

shangxinli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

manuzhang commented Oct 13, 2026 •

edited

Loading