[Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and Hive to 4.0.1 for thriftserver by nileshrathi345 · Pull Request #567 · apache/livy

nileshrathi345 · 2026-10-12T17:21:59Z

What changes were proposed in this pull request?

This PR upgrades the Livy Thrift Server dependency on org.apache.thrift:libthrift from 0.9.3 to 0.25.0 (LIVY-1091).

Motivation: Apache Thrift before 0.23.0 is affected by CVE-2026-43869 (TLS hostname verification in TSSLTransportFactory). Livy’s Thrift Server uses libthrift for binary and HTTP CLI/JDBC transport; bumping to 0.25.0 addresses that class of issues and removes a very old pinned dependency.

Jira: https://issues.apache.org/jira/browse/LIVY-1091

This PR also bumps hive.version from 3.0.0 to 4.0.1 for the Thrift server and related modules. It builds on LIVY-1091 (libthrift 0.25.0 for GHSA-7pwc-h2j2-rjgj). Hive 3.0.0 was compiled against libthrift 0.9.3; Hive 4.0.1 uses libthrift 0.16.0 and Java 8, which fits Livy's -Pspark3 / JDK 8 CI leg while Livy continues to run libthrift 0.25 on the server classpath.

Hive 4 API compatibility: commons-lang3 usage, PasswdAuthenticationProvider.Authenticate (capital A in 4.0.1), and stubs for UploadData / DownloadData. Response structs require operationHandle; a placeholder handle is set with the unsupported status so Thrift serialization succeeds and the client connection is not dropped.
Hive 4 JDBC packaging: hive-jdbc:standalone for the beeline client module. For thriftserver and integration-test, the thin hive-jdbc jar is supplemented with test-scoped hive-common and hive-serde (Hive 4.x does not pull these via Maven transitives). Do not use standalone on the server IT classpath—it relocates libthrift and conflicts with compile-scoped hive-service-rpc.
HTTP integration tests: Test-only THttpClient with the HttpClient 4.x constructor expected by Hive JDBC when the IT JVM loads org.apache.thrift. The production HTTP server uses ThriftHttpServlet, not libthrift THttpClient.

JIRA: https://issues.apache.org/jira/browse/LIVY-1093

Why one PR for two Jiras: LIVY-1091 (libthrift 0.25.0) and LIVY-1093 (Hive 4.0.1) are coupled for the Thrift server: Hive 3.0.0 was built against libthrift 0.9.3, while Livy must move to libthrift 0.25.0 for CVE-2026-43869. Upgrading Hive to 4.0.1 keeps service RPC, JDBC, and authentication aligned with a supported Hive line and avoids splitting two dependent upgrades across separate PRs that would not build or test cleanly in isolation.

How was this patch tested?

Compile (Spark 3):
mvn -Pthriftserver -Pspark3 -Pscala-2.12 -pl thriftserver/server,thriftserver/session -am compile -DskipTests
Compile (Spark 4 / default):
mvn -Pthriftserver -pl thriftserver/server,thriftserver/session -am compile -DskipTests
Unit tests:
mvn -Pthriftserver -Pspark3 -Pscala-2.12 -pl thriftserver/session test

livy-thriftserver: unit and integration tests, including new coverage for TLS (binary and HTTP), UploadData / DownloadData, max message size, and LibthriftHttpCompatSuite.
(ColumnBufferTest, 6 tests)

Was this patch authored or co-authored using generative AI tooling?

Co-authored by: Cursor AI (Claude Opus 4.8)

nileshrathi345 · 2026-10-12T17:26:09Z

@gyogal and @roczei
could you please take a look when possible and review
thanks..!!!!

roczei · 2026-10-12T19:31:45Z

Hi @nileshrathi345 ,

Thanks for the PR! Unfortunately there is a CI failure, could you please fix it? Also, please double check locally if all unit and integration tests run without errors for both Spark 4 and Spark 3 using the following commands:

mvn -Pthriftserver -Pspark3 -Pscala-2.12 -B  verify
mvn -Pthriftserver -B verify

nileshrathi345 · 2026-10-13T07:38:39Z

Hi @nileshrathi345 ,

Thanks for the PR! Unfortunately there is a CI failure, could you please fix it? Also, please double check locally if all unit and integration tests run without errors for both Spark 4 and Spark 3 using the following commands:
mvn -Pthriftserver -Pspark3 -Pscala-2.12 -B  verify
mvn -Pthriftserver -B verify

Hi @roczei — thanks for the review.

Why CI failed

After moving to libthrift 0.25, HttpThriftServerSuite failed with:

NoSuchMethodError: THttpClient.(String, org.apache.http.client.HttpClient)

The HTTP thrift integration tests connect with Hive JDBC 3.0, which still builds the HTTP transport using Apache HttpClient 4.x and that constructor.

libthrift 0.25 still has THttpClient, but it only supports HttpClient 5.x — the HttpClient 4 constructor was removed. So the class is present, but the method Hive JDBC calls is not → runtime failure when the suite opens a JDBC connection in HTTP mode.

This is on the test client (JDBC driver) side. The Livy Thrift HTTP server uses our ThriftHttpServlet path, not libthrift’s THttpClient.

Fix:

Added a test-only org.apache.thrift.transport.THttpClient under thriftserver/server/src/test/java that restores the HttpClient 4.x constructor for the JDBC client during ITs. It is not part of the shipped server artifact.

Local verification

Both commands completed successfully on my side:

mvn -Pthriftserver -Pspark3 -Pscala-2.12 -B verify
mvn -Pthriftserver -B verify

nilesh.rathi@HQV9WP4F7Y livy % mvn -Pthriftserver -B verify
 [INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 1 licenses.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Livy Project Parent POM 1.0.0-SNAPSHOT:
[INFO]
[INFO] Livy Project Parent POM ............................ SUCCESS [ 3.865 s]
[INFO] livy-api ........................................... SUCCESS [ 7.253 s]
[INFO] livy-client-common ................................. SUCCESS [ 2.935 s]
[INFO] livy-test-lib ...................................... SUCCESS [ 3.826 s]
[INFO] multi-scala-project-root ........................... SUCCESS [ 0.992 s]
[INFO] livy-core-parent ................................... SUCCESS [ 0.309 s]
[INFO] livy-core_2.13 ..................................... SUCCESS [ 1.688 s]
[INFO] livy-rsc ........................................... SUCCESS [ 31.163 s]
[INFO] livy-repl-parent ................................... SUCCESS [ 3.981 s]
[INFO] livy-repl_2.13 ..................................... SUCCESS [01:36 min]
[INFO] livy-server ........................................ SUCCESS [03:42 min]
[INFO] livy-thriftserver-session .......................... SUCCESS [ 13.825 s]
[INFO] livy-thriftserver .................................. SUCCESS [05:44 min]
[INFO] livy-assembly ...................................... SUCCESS [ 5.076 s]
[INFO] livy-client-http ................................... SUCCESS [ 13.519 s]
[INFO] livy-scala-api-parent .............................. SUCCESS [ 1.303 s]
[INFO] livy-scala-api_2.13 ................................ SUCCESS [ 21.939 s]
[INFO] livy-integration-test .............................. SUCCESS [25:58 min]
[INFO] livy-coverage-report ............................... SUCCESS [ 3.833 s]
[INFO] livy-examples ...................................... SUCCESS [ 4.560 s]
[INFO] livy-python-api .................................... SUCCESS [ 2.868 s]
[INFO] livy-beeline ....................................... SUCCESS [ 3.500 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39:09 min
[INFO] Finished at: 2026-10-13T12:25:15+05:30
[INFO] ------------------------------------------------------------------------
nilesh.rathi@HQV9WP4F7Y livy %   nilesh.rathi@HQV9WP4F7Y livy % mvn -Pthriftserver -Pspark3 -Pscala-2.12 -B verify  [INFO] Rat check: Summary over all files. Unapproved: 0, unknown: 0, generated: 0, approved: 1 licenses.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Livy Project Parent POM 1.0.0-SNAPSHOT:
[INFO]
[INFO] Livy Project Parent POM ............................ SUCCESS [ 3.367 s]
[INFO] livy-api ........................................... SUCCESS [ 5.516 s]
[INFO] livy-client-common ................................. SUCCESS [ 3.226 s]
[INFO] livy-test-lib ...................................... SUCCESS [ 3.005 s]
[INFO] multi-scala-project-root ........................... SUCCESS [ 0.946 s]
[INFO] livy-core-parent ................................... SUCCESS [ 0.353 s]
[INFO] livy-core_2.12 ..................................... SUCCESS [ 1.505 s]
[INFO] livy-rsc ........................................... SUCCESS [ 26.984 s]
[INFO] livy-repl-parent ................................... SUCCESS [ 2.992 s]
[INFO] livy-repl_2.12 ..................................... SUCCESS [01:51 min]
[INFO] livy-server ........................................ SUCCESS [03:33 min]
[INFO] livy-thriftserver-session .......................... SUCCESS [ 12.268 s]
[INFO] livy-thriftserver .................................. SUCCESS [06:01 min]
[INFO] livy-assembly ...................................... SUCCESS [ 3.838 s]
[INFO] livy-client-http ................................... SUCCESS [ 12.382 s]
[INFO] livy-scala-api-parent .............................. SUCCESS [ 0.662 s]
[INFO] livy-scala-api_2.12 ................................ SUCCESS [ 21.920 s]
[INFO] livy-integration-test .............................. SUCCESS [19:46 min]
[INFO] livy-coverage-report ............................... SUCCESS [ 3.499 s]
[INFO] livy-examples ...................................... SUCCESS [ 4.140 s]
[INFO] livy-python-api .................................... SUCCESS [ 2.674 s]
[INFO] livy-beeline ....................................... SUCCESS [ 3.465 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:05 min
[INFO] Finished at: 2026-10-13T11:43:35+05:30
[INFO] ------------------------------------------------------------------------
nilesh.rathi@HQV9WP4F7Y livy %

codecov-commenter · 2026-10-13T14:39:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.75%. Comparing base (fc5f238) to head (cc4c604).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@              Coverage Diff              @@
##             master     #567       +/-   ##
=============================================
- Coverage     69.03%   52.75%   -16.28%     
+ Complexity     1231      853      -378     
=============================================
  Files           107      107               
  Lines          6986     6979        -7     
  Branches        858      857        -1     
=============================================
- Hits           4823     3682     -1141     
- Misses         1689     2849     +1160     
+ Partials        474      448       -26

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

roczei

LGTM

Bump libthrift to 0.25.0 (CVE-2026-43869). Adapt thriftserver for libthrift 0.25 APIs and add test-only THttpClient (HttpClient 4.x) so HttpThriftServerSuite passes with Hive JDBC on the test classpath.

gyogal · 2026-10-21T20:30:32Z

Thanks for your PR @nileshrathi345! Could you please use a single JIRA and use that as the PR title? The commits will be squashed during the merge and they should be in the usual format.

Bump hive.version from 3.0.0 to 4.0.1 so the Thrift server aligns with libthrift 0.25.0 (LIVY-1091). Hive 3.0.0 was built for libthrift 0.9.3; Hive 4.0.1 targets libthrift 0.16.0 and Java 8. - Adapt to Hive 4 APIs: commons-lang3, PasswdAuthenticationProvider.Authenticate, UploadData/DownloadData stubs with placeholder operationHandle. - hive-jdbc:standalone for beeline; hive-common/hive-serde at test scope for IT. - Test-only THttpClient (HttpClient 4.x); TLS and protocol IT coverage.

nileshrathi345 force-pushed the LIVY-1091 branch from 7509144 to 3c6eb8f Compare October 12, 2026 17:23

roczei approved these changes Oct 13, 2026

View reviewed changes

gyogal mentioned this pull request Oct 19, 2026

[LIVY-1093][THRIFT] Upgrade Hive to 4.0.1 for thriftserver #568

Closed

nileshrathi345 force-pushed the LIVY-1091 branch from 1fd8412 to fac4178 Compare October 21, 2026 04:52

[LIVY-1091] Upgrade libthrift 0.9.3 to 0.25.0

968b6ab

Bump libthrift to 0.25.0 (CVE-2026-43869). Adapt thriftserver for libthrift 0.25 APIs and add test-only THttpClient (HttpClient 4.x) so HttpThriftServerSuite passes with Hive JDBC on the test classpath.

nileshrathi345 force-pushed the LIVY-1091 branch from fac4178 to 968b6ab Compare October 21, 2026 05:06

nileshrathi345 changed the title ~~[Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0~~ [Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and [LIVY-1093] Upgrade Hive to 4.0.1 for thriftserver Oct 21, 2026

gyogal reviewed Oct 21, 2026

View reviewed changes

Comment thread ...ftserver/server/src/main/scala/org/apache/livy/thriftserver/cli/ThriftBinaryCLIService.scala

gyogal requested changes Oct 21, 2026

View reviewed changes

nileshrathi345 changed the title ~~[Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and [LIVY-1093] Upgrade Hive to 4.0.1 for thriftserver~~ [Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and Hive to 4.0.1 for thriftserver Oct 22, 2026

nileshrathi345 force-pushed the LIVY-1091 branch from cc4c604 to 46005e1 Compare October 22, 2026 05:38

nileshrathi345 force-pushed the LIVY-1091 branch from 46005e1 to 5417a76 Compare October 22, 2026 05:45

Navigation Menu

Repository navigation

[Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and Hive to 4.0.1 for thriftserver - #567

[Livy-1091] Upgrade libthrift 0.9.3 to 0.25.0 and Hive to 4.0.1 for thriftserver#567
nileshrathi345 wants to merge 2 commits into
apache:masterfrom
nileshrathi345:LIVY-1091

nileshrathi345 commented Oct 12, 2026 •

edited

Loading

Uh oh!

nileshrathi345 commented Oct 12, 2026

Uh oh!

roczei commented Oct 12, 2026

Uh oh!

nileshrathi345 commented Oct 13, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 13, 2026 •

edited

Loading

Uh oh!

roczei left a comment

Uh oh!

Uh oh!

gyogal commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

nileshrathi345 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

nileshrathi345 commented Oct 12, 2026

Uh oh!

roczei commented Oct 12, 2026

Uh oh!

nileshrathi345 commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

roczei left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gyogal commented Oct 21, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nileshrathi345 commented Oct 12, 2026 •

edited

Loading

nileshrathi345 commented Oct 13, 2026 •

edited

Loading

codecov-commenter commented Oct 13, 2026 •

edited

Loading