HIVE-25996: Backport HIVE-25098 to branch-2.3

[wangyum]

What changes were proposed in this pull request?

Backport HIVE-25098 to branch-2.3.

Why are the changes needed?

Make the downstreams easy to upgrade their Thrift version.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Local test.

[dongjoon-hyun]

Just a question: What about branch-3.1 and branch-3.0? Are these patches released officially in any Apache Hive artifacts after vote?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.

[wangyum]

cc @sunchao

[pan3793]

Thanks to the work done by @wangyum, Hive 2.3 is widely used in the industry and adopted by many downstream projects such as Apache Spark. A security patch release is greatly appreciated.

[bgmarsh]

Are there still plans to get this into the Hive 2.3 branch?

[pan3793]

cc @sunchao this is the first step of upgrading thrift to get rid of CVE (0.14.0+), as we are planning Hive 2.3.10, please reopen this PR.

[sunchao]

@wangyum could you rebase the PR?

[wangyum]

This file removed by: 1945e2f

[wangyum]

This file removed by: 1945e2f

[pan3793]

There are around 30~40 tests failure in the latest branch-2.3, some of them are flaky.
36, 28, 34 test failures were reported by the latest three commits.

Jenkins reports 41 test failures for this PR, which seems reasonable.

[pan3793]

Verified thrift-generated files locally with

# install thrift compiler 0.14.1 at /home/chengpan/bin/thrift-0.14.1/compiler/cpp/cmake-build
# provide share/fb303/if/fb303.thrift
mvn -pl metastore -Pthriftif -Dthrift.home=/home/chengpan/bin/thrift-0.14.1/compiler/cpp/cmake-build compile -am

Without fb303.thrift the compile command failed with the following error message, is it expected?

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-thrift-sources) on project hive-metastore: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /home/chengpan/Projects/hive-2.3/metastore/target/antrun/build-main.xml:15: exec returned: 1
[ERROR] around Ant part ...<for param="thrift.file">... @ 9:28 in /home/chengpan/Projects/hive-2.3/metastore/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hive-metastore

[wangyum]

@pan3793 It should throw Could not find include file share/fb303/if/fb303.thrift:

root@e8c54f097cdf:/opensource/HIVE-25996# mvn -pl metastore -Pthriftif -DskipTests=true package -Dthrift.home=/usr/local
[INFO] Scanning for projects...
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] Building Hive Metastore 2.3.10-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-metastore ---
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-banned-dependencies) @ hive-metastore ---
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-property) @ hive-metastore ---
[INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-thrift-sources) @ hive-metastore ---
[INFO] Executing tasks
main:
[delete] Deleting directory /opensource/HIVE-25996/metastore/src/gen/thrift
[mkdir] Created dir: /opensource/HIVE-25996/metastore/src/gen/thrift
[echo] Generating Thrift code for /opensource/HIVE-25996/metastore/if/hive_metastore.thrift
[exec] [ERROR:/opensource/HIVE-25996/metastore/if/hive_metastore.thrift:1033] (last token was 'fb303.FacebookService')
[exec] Service "fb303.FacebookService" has not been defined.
[exec] [WARNING:/opensource/HIVE-25996/metastore/if/hive_metastore.thrift:25] Could not find include file share/fb303/if/fb303.thrift
[exec] [INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.199 s
[INFO] Finished at: 2023-07-24T12:59:31+00:00
[INFO] Final Memory: 33M/362M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (generate-thrift-sources) on project hive-metastore: An Ant BuildException has occured: The following error occurred while executing this line:
[ERROR] /opensource/HIVE-25996/metastore/target/antrun/build-main.xml:15: exec returned: 1
[ERROR] around Ant part ...<for param="thrift.file">... @ 9:28 in /opensource/HIVE-25996/metastore/target/antrun/build-main.xml
[ERROR] -> [Help 1]
[ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

It works if add fb303.thrift to share/fb303/if/:

root@e8c54f097cdf:/opensource/HIVE-25996# mvn -pl metastore -Pthriftif -DskipTests=true package -Dthrift.home=/usr/local
[INFO] Scanning for projects...
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] Building Hive Metastore 2.3.10-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @ hive-metastore ---
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-banned-dependencies) @ hive-metastore ---
[INFO] [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-property) @ hive-metastore ---
[INFO] [INFO] --- maven-antrun-plugin:1.7:run (generate-thrift-sources) @ hive-metastore ---
[INFO] Executing tasks
main:
[delete] Deleting directory /opensource/HIVE-25996/metastore/src/gen/thrift
[mkdir] Created dir: /opensource/HIVE-25996/metastore/src/gen/thrift
[echo] Generating Thrift code for /opensource/HIVE-25996/metastore/if/hive_metastore.thrift
[INFO] Executed tasks
[INFO] ...

[pan3793]

Yes, I fixed it by adding share/fb303/if/fb303.thrift, so why share/fb303/if/fb303.thrift is not included in Hive source?

[sunchao]

LGTM.

I manually verified the Thrift changes, and the CI results. There is one new test failure after this but it is unrelated.

This PR is important to fix a CVE issue in branch-2.3

[sunchao]

@wangyum could you resolve the conflict? I'll merge this after it.

[wangyum]

Thank you @sunchao. Conflict resolved.

[sunchao]

Thanks! merged to branch-2.3

[Madhukar525722]

Hi @wangyum@pan3793@sunchao
There is need to backport - HIVE-26633 as well along with this. The limit of 100mb from libthrift client side is very less for a large table. Causing query failures from spark end. Related issue - https://issues.apache.org/jira/browse/SPARK-49489

[wangyum]

We have set max message size to Integer.MAX_VALUE for internal Hive: