feat(inspect): add task-based scans for snapshots and history by wgtmac · Pull Request #1066 · apache/iceberg-cpp

wgtmac · 2026-10-27T05:50:13Z

Other Iceberg implementations

Java: Metadata tables reuse TableScan configuration where applicable. Static tables materialize rows into StaticDataTask; other tables use tasks that read manifests or manifest lists. Metadata tasks participate in the file-scan task hierarchy and have JSON parsers.
Go: Table.Inspect() exposes methods for individual metadata tables and returns Arrow record readers. Snapshots and history are materialized into a single batch.
Rust: table.inspect() exposes typed metadata-table objects with schemas and an async scan() returning an Arrow record-batch stream. The inspected snapshots, history, and manifests implementations build a batch before wrapping it in a stream.
PyIceberg: table.inspect exposes methods returning materialized PyArrow tables. Some manifest/file inspection paths use an executor to parallelize reads before collecting the results.

Design decisions

Attempted design	Trade-off / rationale	Decision
Separate metadata scan configuration, task planning, and task reading	Supports individual task consumption and a convenient `ToArrowArrayStream()` path. The builder selects output columns; the reader controls execution settings.	Keep
Reuse `TableScanBuilder` and inherit from the ordinary data-file task hierarchy	Would bring data-file scan contracts into an inspection API whose supported tables need neither data-file splits nor delete-file handling.	Drop
Require Java-compatible task JSON and cross-process task execution	Adds a wire-format and row-representation contract beyond the current local inspection scope.	Drop
Store intermediate rows as generic `StructLike` values or table-specific structs in a central variant	Duplicates data already present in metadata and makes each new table extend a shared row registry. Partition values and readable metrics also have schema-dependent structure.	Drop
Store owned Arrow arrays in the task, using ownership wrappers or deep copies for repeated reads	Couples task lifetime to execution buffers and complicates reuse. A reader can build independent batches from the retained source instead.	Drop
Retain shared ownership of the captured `TableMetadata` for snapshots/history tasks	Avoids a second row representation. Built scans retain their configuration and captured metadata across builder changes and normal table refreshes; mutable aliases must not modify the retained metadata.	Keep
Produce one reusable task for snapshots/history	Splitting snapshots/history adds scheduling overhead without useful I/O parallelism. Each reader maintains independent consumption state.	Keep
Materialize every Arrow batch when opening a task	Allocates the full output before consumption and performs unnecessary work when reading stops early.	Drop
Generate Arrow batches lazily in the reader	Limits producer-owned Arrow output to the batch being built and avoids converting unread rows. Source metadata and auxiliary indexes remain retained.	Keep
Configure Arrow batch size in the scan builder	Batch boundaries are a reading concern and do not affect the planned work. Batch size belongs in `MetadataScanTaskReader::Options`.	Drop
Implement concrete scans and column conversion in their corresponding table files	Keeps table-specific schemas and behavior together. Common utilities handle batch construction, projection validation, and stream ownership.	Keep
Publish options, filters, time travel, executor controls, or concrete task classes before implementing them	Creates unsupported or ineffective public configuration and prematurely fixes extension contracts.	Drop
Expose an estimated row count on metadata tasks	Has no current scheduling or execution consumer; it can be introduced when a concrete use requires it.	Drop

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

- Add Table::Inspect() and metadata scan builders for column selection, projection, and case-sensitive name resolution. - Separate task planning from reading with reusable static tasks backed by captured table metadata. - Preserve scan configuration and metadata across builder changes and table refreshes. - Produce Arrow batches lazily through MetadataScanTaskReader, with reader-controlled batch sizes and independent stream lifetimes. - Keep canonical schemas and row conversion in each metadata table implementation, sharing batch construction and projection validation. - Add common Arrow table comparison utilities and tests for schemas, full reads, reordered projections, and stream behavior

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

manuzhang · 2026-10-27T11:24:58Z

+        return std::nullopt;
+      }
+      return Result<std::optional<T>>(std::in_place, std::in_place,
+                                      std::move_if_noexcept(values_[index_++]));


this doesn't work with std::vector<bool> which returns a temporary proxy object for values_[index_++]

manuzhang · 2026-10-27T11:40:11Z

+  /// The returned stream owns its read state and can outlive this scan. The caller
+  /// must release the stream and each returned schema or array independently.
+  /// Releasing the stream early closes the active task stream. To customize
+  /// batch size, open planned tasks with MetadataScanTaskReader instead.


It looks there's no public API for MetadataScanTaskReader to access internal fields of MetadataScanContext.

manuzhang · 2026-10-27T13:42:38Z

+      new HistoryScan(std::move(context), std::move(task)));
+}
+
+Result<MetadataScanTaskStreamPtr> HistoryScan::PlanTasksStream() const {


The follow methods look same as SnapshotTable's. Can they be moved to superclass?

manuzhang · 2026-10-27T13:57:28Z

+struct MetadataScanContext {
+  MetadataTableKind kind;
+  std::shared_ptr<const TableMetadata> table_metadata;
+  std::string metadata_file_location;


How are metadata_file_location and io supposed to be used? It looks they are unused today.

I guess these will be used by other metadata table kinds, such as AllManifestsTable, AllEntriesTable and AllFilesTable etc. in java impl.

manuzhang · 2026-10-27T14:01:18Z

+  EXPECT_EQ(*metadata_table_->schema(), *expected);
 }

-TEST_F(SnapshotsTableTest, ScanEmptySnapshotList) {


Do we have the same coverage as before?

manuzhang · 2026-10-27T14:03:11Z

@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one


Our convention is to use /* */ blocks for license headers if you check existing files.

zhjwpku · 2026-10-27T15:48:13Z

+  field_ids.reserve(output_schema->fields().size());
+  for (const auto& field : output_schema->fields()) {
+    ICEBERG_ASSIGN_OR_RAISE(auto canonical_field,
+                            canonical_schema->FindFieldById(field.field_id()));


Should we use GetFieldById() here to check only top-level fields? FindFieldById() also finds nested fields, so a projection with summary.key (field ID 7) as a top-level field passes validation.

Make the scan surface a subset of the API proposed in apache#1066 so the two can converge: - MetadataTable::NewScan() is virtual and const, and ScanProjection() is const. - MetadataScanBuilder::Select() resolves names immediately with the current CaseSensitive() setting and records errors for Build(); Project() sets the output schema directly, and Build() rejects a null projection. - MetadataScan::ToArrowArrayStream() is virtual and const. - Projection validation rejects unknown columns, type changes, and projecting a nullable column as required, with the same messages as apache#1066. - Drop the time-travel options together with TimeTravelMetadataTable and SnapshotSelection, which apache#1066 also removes; nothing on main implements them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…Build() Drop the MetadataScan class: separating a built scan from the Arrow stream it reads belongs to the task-based design in apache#1066. Build() now validates the options and returns the stream directly, and each call returns a new stream. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

shangxinli · 2026-10-28T17:56:16Z

+    const std::shared_ptr<Schema>& projected_schema, int64_t batch_size) {
+  std::vector<size_t> snapshot_indices;
+  for (size_t i = 0; i < metadata->snapshots.size(); ++i) {
+    if (metadata->snapshots[i]) {


nit: no test exercises this skip when an entry in metadata->snapshots is null.

shangxinli · 2026-10-28T17:56:18Z

+            return operation ? AppendString(column, *operation) : AppendNull(column);
+          }
+          case 5:  // manifest_list
+            return snapshot.manifest_list.empty()


nit: no test exercises the empty manifest_list branch here.

shangxinli · 2026-10-28T17:56:20Z

+  // Walk the current snapshot's ancestry; the insert check stops on cycles.
+  std::unordered_set<int64_t> ancestors;
+  for (auto current = metadata->current_snapshot_id;
+       current != kInvalidSnapshotId && ancestors.insert(current).second;) {


nit: the cycle guard here isn't tested against an actual cycle in the snapshot graph, so it's defensive-only right now.

Copilot AI balanced review requested due to automatic review settings October 27, 2026 05:50

Copilot AI reviewed Oct 27, 2026

wgtmac force-pushed the metadata_table_api_refactor branch from a0fb6c8 to 346d3b7 Compare October 27, 2026 06:00

Copilot AI balanced review requested due to automatic review settings October 27, 2026 06:00

Copilot AI reviewed Oct 27, 2026

manuzhang reviewed Oct 27, 2026

View reviewed changes

zhjwpku reviewed Oct 27, 2026

View reviewed changes

shangxinli reviewed Oct 28, 2026

View reviewed changes

		@@ -0,0 +1,44 @@
		// Licensed to the Apache Software Foundation (ASF) under one

Conversation

wgtmac commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Other Iceberg implementations

Design decisions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wgtmac commented Oct 27, 2026 •

edited

Loading