Repository navigation
Description
Describe the bug
When an Iceberg table has the write.data.path property set, ALTER TABLE ... EXPORT PART and EXPORT PARTITION ignore it. The exported data files are written under <table location>/data/ instead of the configured path. A regular INSERT into the same table does follow the property, so a table that gets both kinds of writes ends up with its data split across two locations.
The table stays valid and readable, because the manifests store the full path of every data file. The issue is where the files end up. Anything that expects new data under write.data.path won't see the exported files, such as lifecycle rules, prefix-based access policies or cleanup jobs. If the writer's credentials only allow writes under the data path, for example vended or prefix-scoped catalog credentials, the export may fail with access denied while an INSERT into the same table succeeds.
The cause is that write.data.path is only applied in the IcebergStorageSink constructor, which reads it from the table metadata and calls filename_generator.setDataLocation. IcebergImportSink, which handles EXPORT, creates its own FileNamesGenerator and passes it to MultipleFileWriter without that step. As a result, generateDataFileName falls back to the default <table location>/data/ prefix.
To Reproduce
Steps to reproduce the behavior:
- Create an Iceberg table and set its data path property, for example with Spark:
ALTER TABLE db.tbl SET TBLPROPERTIES ('write.data.path' = 's3://bucket/custom-data-path'). - In ClickHouse, create a MergeTree table with a matching schema, insert some rows and note the part name.
- Export the part into the Iceberg table:
ALTER TABLE mt EXPORT PART 'all_1_1_0' TO TABLE iceberg_tbl
SETTINGS allow_experimental_export_merge_tree_part = 1, allow_experimental_insert_into_iceberg = 1
- Insert a row with a regular
INSERT INTO iceberg_tbl .... - Run
SELECT DISTINCT _path FROM iceberg_tbl. The row from step 4 is in a file unders3://bucket/custom-data-path/, while the exported rows are in files under<table location>/data/.
Expected behavior
EXPORT PART and EXPORT PARTITION should write data files under the location given by write.data.path when it's set, the same as INSERT. When it isn't set, they should keep using <table location>/data/.
Screenshots
N/A
Key information
Provide relevant runtime details.
- Project Antalya Build Version: antalya-26.8, antalya-26.6, and antalya-26.3
- Cloud provider: not provider specific
- Kubernetes provider: not specific
- Object storage: any (AWS S3, MinIO)
- Iceberg catalog: any catalog that allows setting write.data.path (REST, Glue, Unity)
Additional context
The relevant code is in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp:
- antalya-26.8: the property is applied at line 1532 in IcebergStorageSink, and the generator is created without it at line 2058 in IcebergImportSink
- antalya-26.6: applied at line 1488, missing at line 1991
- antalya-26.3: applied at line 1111 (in Antalya 26.3 Backport of #98162 - Improve iceberg writes for unity catalog #2714), missing at line 1589
Activity