[xieandrew]

Closes #2182.

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Improve iceberg write compatibility for Unity and other REST catalogs. Fixes writing a corrupted manifest list when the existing manifest list written externally has a key_metadata field.

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Unit tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• CAS (content-addressed storage; Antalya only)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [4af4c5a]

[xieandrew]

@blau-ai

[blau-ai]

CI triage for 11d1a51

Verdict: 1 failing check, 0 caused by this PR. Stateless tests (amd_debug, parallel) fails on 5 test timeouts. They are on dataset and network-heavy tests that have nothing to do with Iceberg, and the same tests are timing out right now on unrelated antalya-26.3 backports. All other checks marked "skipping" in the GitHub UI were dropped because of this one failure (Dropped due to previous failure [Stateless tests (amd_debug, parallel)]) or excluded by the arm/tsan/msan tag filter. They didn't run and didn't fail.

Check	Result	Classification
Stateless tests (amd_debug, parallel)	5 tests timed out (each one failed in both the main run and the retry, giving the "Failed: 10" count)	Infra / pre-existing on base, not PR-related
Build (amd_debug/asan/ubsan/binary/release, arm_release), Fast test, Quick functional, Stateless amd_asan 1–4/4, Unit tests asan/ubsan, Docker images	✅ pass	—
Integration, Stress, AST fuzzer, BuzzHouse, other stateless suites	⏭ dropped after the failure above	Not run, so they say nothing about this PR yet

Stateless tests (amd_debug, parallel)

Failing tests (report):
04033_tpc_ds_q09, 04033_tpc_ds_q14, 03634_autopr_output_bytes_estimation, 00152_insert_different_granularity, 00154_avro. All hit the 600 s test timeout or the client's 300 s receive timeout. In the first pass, 04033_tpc_ds_q24/q34/q67/q23 and 03634_autopr_input_bytes_estimation also timed out but passed on retry.

Why it isn't the PR: the hung-query check at the end of job.log shows queries such as SELECT * FROM test.hits WHERE URL LIKE '%google%' ... stuck inside DNS resolution during a TLS connect:

clickhouse::__poll
clickhouse::__GI___res_context_send
clickhouse::_nss_dns_gethostbyname4_r
clickhouse::getaddrinfo
Poco::Net::DNS::hostByName(...)                      base/poco/Net/src/DNS.cpp:80
Poco::Net::SecureSocketImpl::isLocalHost(...)
Poco::Net::SecureSocketImpl::verifyPeerCertificateImpl(...)
...
DB::EndpointConnectionPool<Poco::Net::HTTPSClientSession>::prepareNewConnection(...)

• The tests that timed out all read remote datasets over HTTPS (test.hits, TPC-DS tables, avro and granularity fixtures). The runner's DNS stalled and these queries waited until the timeout. That is an infra/network symptom.
• The PR only touches Iceberg write paths, RestCatalog.cpp, and a single blank line in IKeeper.h. None of these tests exercise those paths.
• The same failure shows up on unrelated sibling PRs on the same base today:
  • Antalya 26.3 Backport of #102156 - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN #2712 (Nullable ALTER fix): Stateless tests (amd_debug, parallel) failed with 13 failures: 04033_tpc_ds_q09/q14/q24, 00154_avro, 03634_autopr_output_bytes_estimation, 00152_insert_different_granularity, … (a superset of ours).
  • Antalya 26.3 Backport of #127318 - Distrust the compromised eMudhra certificates #2697 (eMudhra certificates): failed with 04033_tpc_ds_q09/q14, 03634_autopr_input_bytes_estimation, 00071_merge_tree_optimize_aio.
• The 2 BROKEN tests (03441_deltalake_*) are expected-broken and don't count toward the failure.

Next step: re-run the failed jobs. No code change is needed on this PR. If it times out again, the problem is the runner's DNS/egress on the antalya-26.3 stateless runners and needs someone with CI access, not a change to this backport. I found no open tracking issue for it, so filing one may be worth it, given that at least three PRs are affected.

Caveat: because of this failure, the integration tests (where the Iceberg/REST-catalog coverage lives), the stress tests and the fuzzers have not run on this PR. Don't take the green checks as proof the backport is correct until a re-run gets past this stage.

_{Evidence: praktika result_pr.json and job.log for 11d1a51, plus result_pr.json for #2712 (ede57c1) and #2697 (3c607d2).}

[subkanthi]

Medium — write.data.path only on IcebergStorageSink
Yes, from this PR’s change. The PR adds setDataLocation / write.data.path and only wires it in IcebergStorageSink. IcebergImportSink was left untouched, so the new property is applied inconsistently. Same gap exists in upstream #98162.

Low — stale TODO above fail-after-retries
Caused by this PR’s edit. The fail-closed throw is new; the old “perhaps we should fail” TODO was not removed/updated, so the comment became wrong.

@xieandrew if these are valid maybe we can create separate issues to track these.

[xieandrew]

@subkanthi I created an issue for write.data.path specifically for EXPORT PART/PARTITION here #2757. This issue still exists in 26.6 and 26.8. There are other instances of this but they affect non-Antalya upstream as mentioned.

For the second issue the comment is still on other branches like 26.6 and 26.8. Removing it in this PR would deviate from other branches for no functional difference.

[xieandrew]

@blau-ai

[blau-ai]

CI triage for 4af4c5a

Verdict: 6 failing checks. 0 caused by this PR, 4 pre-existing or infra, 2 timeouts with no evidence pointing at the PR (re-run them).

Check	Failing test(s)	Class	PR-caused?
Stateless tests (amd_debug, sequential)	03271_s3_table_function_asterisk_glob	pre-existing (external network)	❌
Stateless tests (amd_debug, distributed plan, s3 storage, sequential)	03271_s3_table_function_asterisk_glob	pre-existing (external network)	❌
Integration tests (amd_asan, db disk, old analyzer, 7/8)	test_scheduler_query::{test_create, test_max_concurrent_queries, test_max_waiting_queries_reached}	infra / test config	❌
Regression release aes_encryption	/aes encryption/decrypt_mysql/key or iv length for mode/* (6 examples)	pre-existing	❌
GrypeScanServer / Grype Scan	none (job timed out)	infra timeout	❌
RegressionTestsRelease / LightweightDelete (3)	unknown (job timed out, no log)	timeout, probably infra	very unlikely, re-run

This PR only changes Iceberg write code (src/Storages/ObjectStorage/DataLakes/Iceberg/* and RestCatalog.cpp). None of the failures above go through that code.

---

1. 03271_s3_table_function_asterisk_glob (both amd_debug sequential jobs)

Error on processing query: Timeout exceeded while receiving data from server. Waited for 300 seconds
SELECT * FROM s3('https://clickhouse-public-datasets.s3.amazonaws.com/wikistat/original/*', NOSIGN) LIMIT 1

• The test lists a public upstream S3 bucket over the internet.
• The same test fails on the latest antalya-26.3 MasterCI run (400fdb6, run 41553122541) in the same two jobs, and also in arm_binary, sequential.
• Pre-existing. Nothing to do in this PR.

2. test_scheduler_query (Integration amd_asan 7/8, 3 of 526 tests)

Container roottestschedulerquery-gw0-node-1  Creating
Error response from daemon: Range of CPUs is from 0.01 to 12.00, as there are only 12 CPUs available

• The test asks for cpu_limit=15 (tests/integration/test_scheduler_query/test.py:19, added in 7b6ff05), but this runner has only 12 CPUs. Docker refuses to create the container, so all 3 tests fail in the fixture before any ClickHouse code runs.
• Infra / test config, not this PR. A separate fix would be to lower cpu_limit to ≤ 12 or cap it at the host CPU count. A re-run on a bigger runner should pass.

3. Regression aes_encryption: decrypt_mysql/key or iv length for mode

• 6 examples fail with AssertionError: aes-128/192/256 in ecb and cbc modes.
• Antalya 26.3 Backport of #2330 - Add 409 to the non-retriable HTTP Error list #2739 (an unrelated HTTP-retry backport on the same base) fails the exact same 6 examples, with the same summary "14 features (12 ok, 2 failed)" (job).
• Pre-existing on antalya-26.3. It is a MySQL-compat AES behaviour check and has nothing to do with Iceberg.

4. GrypeScanServer / Grype Scan

• The scan itself finished: the commit status says Grype Scan Completed with 0 high/critical vulnerabilities.
• After that, the Parse grype results step hung and GitHub cancelled the job: The job has exceeded the maximum execution time of 6h0m0s.
• The same job takes about 4 minutes on Antalya 26.3 Backport of #2330 - Add 409 to the non-retriable HTTP Error list #2739 and Antalya 26.3 Backport of #102156 - Fix Nullable(X)->Y cross-type ALTER MODIFY COLUMN #2712.
• Infra hang. Safe to re-run.

5. RegressionTestsRelease / LightweightDelete (3)

• Cancelled with The job has exceeded the maximum execution time of 5h0m0s. GitHub keeps no log for the cancelled step, so I could not identify a failing scenario and can't give a firm verdict.
• The same shard passes in 55 minutes on Antalya 26.3 Backport of #2330 - Add 409 to the non-retriable HTTP Error list #2739.
• The lightweight-delete suite works on MergeTree and doesn't touch any code changed here, so a stuck runner or suite is much more likely than a regression. Re-run it. If it hangs again on this PR only, that would be worth looking into.

Suggested next steps

• Re-run Grype Scan and LightweightDelete (3).
• The rest are known base-branch problems and don't block this backport.
• No code changes are suggested for this PR.

[xieandrew]

No CI fails from this PR. Reran timed out RegressionTestsRelease / LightweightDelete (3) and it passed.

[zvonand]

@xieandrew either add a "verified" label or move it to "Needs verification"

FOr backports, we usually self-verify them

[xieandrew]

@zvonand Added the PR label for self-verified. It was on GH project not PR as well, my mistake.