[EPIC] Fast manifest reads for PyIceberg through pyiceberg-core #3262

What's the feature are you trying to implement?

PyIceberg wants to read manifests and manifest lists through pyiceberg-core instead of its Cython Avro decoder. This issue tracks the iceberg-rust work needed to make that path faster than Cython. The motivation, benchmarks, and PyIceberg tasks are in apache/iceberg-python#4007. This issue falls under the PyIceberg integration epic #1694 and revisits #1280.

Task list

Task 5 needs Tasks 1 to 3 and #3241. Task 4 doesn't block the release, but PyIceberg's scan planning and manifest-list reads need it.

• Task 1: apache-avro 0.22 and a byte-level manifest reader. perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value #3354
• Task 2: Build the pyiceberg-core wheel with opt-level = 3. Build the pyiceberg-core wheel with opt-level = 3 #3646 (PR build(python): build pyiceberg-core wheel with opt-level = 3 #3653)
• Task 3: Read manifests and manifest lists into Arrow with arrow-avro.
  • feat(arrow): read manifest entries into Arrow with arrow-avro #3388
  • feat(arrow): read manifest lists into Arrow with arrow-avro #3631
  • Decide whether the Arrow manifest readers return an iterator of batches #3647 (PR feat(python): read manifests and manifest lists as pyarrow RecordBatchReaders #3652)
  • Return manifests and manifest lists as pyarrow tables from pyiceberg-core #3648, after Decide whether the Arrow manifest readers return an iterator of batches #3647 (PR feat(python): read manifests and manifest lists as pyarrow RecordBatchReaders #3652)
• Task 4: Filter manifests and manifest lists over Arrow. Filter manifests and manifest lists over Arrow #3649
• Raise Python errors instead of panicking in the binding. fix(python): raise instead of panicking in the manifest bindings #3241
• Task 5: Release pyiceberg-core. Release Iceberg Rust (0.12.0) #3628

These items improve iceberg-rust's own readers, Manifest::parse_avro and ManifestList::parse_with_version. PyIceberg won't read through them once Task 3 lands, so they're off this epic's critical path. Most were built during Task 1 but held back to limit it to the behavior changes that come with the upgrade.

• perf: parse manifest list entries without buffering Avro values #3224
• fix(manifest-list): read V1 file counts written with spec field names #3447, which needs perf: parse manifest list entries without buffering Avro values #3224
• fix(spec): match manifest partition fields by field ID #3566, which fixes the read side of avro schema reader ignores iceberg-field-name property — returns sanitized names from java manifests #2536 (avro schema writer does not sanitize field names that violate avro naming rules #2535 covers the write side)
• Match manifest_entry and data_file fields by field-id in Manifest::parse_avro #3650
• Match partition values to the partition spec in deserialize_data_file_from_json #3651
• Write uuid partition values as fixed[16] with logical type uuid, which is how the spec maps uuid to Avro. This changes the file format, so it needs a compatibility check. It overlaps with Partitions with type UUID fail to create transactions #2913 and fix: table UUID partitions not working #2916.
• Remove the workaround for arrow-avro 59 records without fields (fix(arrow-avro): decode records with no fields arrow-rs#10771) from read_manifest_entries once Bump to Arrow 60 #3257 moves to arrow 60.

Willingness to contribute

I can contribute to this feature independently