Skip to content

EXPORT PART/PARTITION to Iceberg ignores the write.data.path table property #2757

Open

Description

@xieandrew

Describe the bug
When an Iceberg table has the write.data.path property set, ALTER TABLE ... EXPORT PART and EXPORT PARTITION ignore it. The exported data files are written under <table location>/data/ instead of the configured path. A regular INSERT into the same table does follow the property, so a table that gets both kinds of writes ends up with its data split across two locations.

The table stays valid and readable, because the manifests store the full path of every data file. The issue is where the files end up. Anything that expects new data under write.data.path won't see the exported files, such as lifecycle rules, prefix-based access policies or cleanup jobs. If the writer's credentials only allow writes under the data path, for example vended or prefix-scoped catalog credentials, the export may fail with access denied while an INSERT into the same table succeeds.

The cause is that write.data.path is only applied in the IcebergStorageSink constructor, which reads it from the table metadata and calls filename_generator.setDataLocation. IcebergImportSink, which handles EXPORT, creates its own FileNamesGenerator and passes it to MultipleFileWriter without that step. As a result, generateDataFileName falls back to the default <table location>/data/ prefix.

To Reproduce
Steps to reproduce the behavior:

  1. Create an Iceberg table and set its data path property, for example with Spark: ALTER TABLE db.tbl SET TBLPROPERTIES ('write.data.path' = 's3://bucket/custom-data-path').
  2. In ClickHouse, create a MergeTree table with a matching schema, insert some rows and note the part name.
  3. Export the part into the Iceberg table:
ALTER TABLE mt EXPORT PART 'all_1_1_0' TO TABLE iceberg_tbl
SETTINGS allow_experimental_export_merge_tree_part = 1, allow_experimental_insert_into_iceberg = 1
  1. Insert a row with a regular INSERT INTO iceberg_tbl ....
  2. Run SELECT DISTINCT _path FROM iceberg_tbl. The row from step 4 is in a file under s3://bucket/custom-data-path/, while the exported rows are in files under <table location>/data/.

Expected behavior
EXPORT PART and EXPORT PARTITION should write data files under the location given by write.data.path when it's set, the same as INSERT. When it isn't set, they should keep using <table location>/data/.

Screenshots
N/A

Key information
Provide relevant runtime details.

  • Project Antalya Build Version: antalya-26.8, antalya-26.6, and antalya-26.3
  • Cloud provider: not provider specific
  • Kubernetes provider: not specific
  • Object storage: any (AWS S3, MinIO)
  • Iceberg catalog: any catalog that allows setting write.data.path (REST, Glue, Unity)

Additional context
The relevant code is in src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions