[HUDI-6509] Add GitHub CI for Java 17 by CTTY · Pull Request #9136 · apache/hudi

CTTY · 2023-07-06T20:41:29Z

Change Logs

Add a new GitHub action to test Hudi spark against Java 17. Some known failures would be excluded as they failed due to Hadoop dependencies like MiniDFSCluster

Impact

None

Risk level (write none, low medium or high below)

None

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

The config description must be updated if new configs are added or the default value of the configs are changed
Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the instruction to make
changes to the website.

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

CTTY · 2023-07-12T22:28:54Z

@hudi-bot run azure

CTTY · 2023-07-18T16:58:24Z

It seems validate-bundles(flink1.17, Spark 3.4, Spark 3.4.0) just consistently fail with JDK17 on issue below:

Connecting to jdbc:hive2://localhost:10000/default
23/07/17 17:45:48 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status.
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0)
Cannot run commands specified using -e. No current connection
Error: validate.sh HiveQL validation failed.
Error: Process completed with exit code 1.

Need to look into this, otherwise everything looks good. newly added docker-test-java17 and test-spark-java17 are working fine

yihua · 2023-07-18T17:01:42Z

+ run:
+ mvn test -Pfunctional-tests -Pjava17 -D"$SCALA_PROFILE" -D"$SPARK_PROFILE" -pl "$SPARK_COMMON_MODULES,$SPARK_MODULES" $MVN_ARGS
+
+ docker-test-java17:


Could this be run with validate-bundles since it already validates bundles on Java 17? Any reason to have a separate job here?

those tests are generally the same as bundle validation but there is one difference: It requires building Hudi within Docker as it runs tests with mvn test command, while bundle validation build Hudi outside of Docker and only copy jars/bundles to Docker. If we consolidates them as one then the job would need to build twice, it would make the job much slower.
If we just seperate them into 2 jobs then docker test can only build hudi-common modules in Docker which is relatively fast and bundle validation can keep the same behavior

I think it's better to have one job and docker setup so it's easier to maintain. We can fix the repeated mvn build later on.

yihua · 2023-07-18T20:12:49Z

+# - <dataset name>/schema.avsc
+# - <dataset name>/data/<data files>
+#################################################################################################
+


Could we consolidate the test logic of this into validate.sh and reuse existing validate-bundle job?

yihua · 2023-07-18T20:15:53Z

 <flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
 <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
- <rocksdbjni.version>5.17.2</rocksdbjni.version>
+ <rocksdbjni.version>7.5.3</rocksdbjni.version>


does RocksDB 5.17.2 not work? Dependency version upgrade has larger impact.

Right, RocksDB 5.17.2 would throw NoClassDefException when running TestHoodieLogFormat
[ERROR] testBasicAppendAndScanMultipleFiles{DiskMapType, boolean, boolean, boolean}[10] Time elapsed: 0.118 s <<< ERROR! 2023-07-13T23:41:36.1420947Z java.lang.NoClassDefFoundError: Could not initialize class org.rocksdb.DBOptions

This seems fishy. Does this only happen for Java 17?

Yes, for Java 8 it works fine. This issue only pops up for Java 17

After reading the code, RocksDbBasedFileSystemView is the only place using RocksDb. So this should be ok.

hudi-bot · 2023-07-24T21:51:26Z

CI report:

a0e7207 UNKNOWN
cda1e77 UNKNOWN
6b33d37 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

This reverts commit 68f6b69.

yihua

LGTM. Thanks for adding validations on Java 17!

yihua · 2023-07-25T18:42:00Z

+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_MODULES: ${{ matrix.sparkModules }}
+ if: ${{ !endsWith(env.SPARK_PROFILE, '3.2') }} # skip test spark 3.2 as it's covered by Azure CI


nit: not required.

yihua · 2023-07-25T18:42:05Z

+ SCALA_PROFILE: ${{ matrix.scalaProfile }}
+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_MODULES: ${{ matrix.sparkModules }}
+ if: ${{ !endsWith(env.SPARK_PROFILE, '3.2') }} # skip test spark 3.2 as it's covered by Azure CI


nit: not required.

yihua · 2023-07-25T18:42:13Z

+ SPARK_PROFILE: ${{ matrix.sparkProfile }}
+ SPARK_RUNTIME: ${{ matrix.sparkRuntime }}
+ SCALA_PROFILE: 'scala-2.12'
+ if: ${{ env.SPARK_PROFILE >= 'spark3.4' }} # Only support Spark 3.4 for now


nit: not required.

yihua · 2023-07-25T18:53:38Z

+if [[ ${SPARK_RUNTIME} == 'spark2.4.8' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=2.3.9
+ DERBY_VERSION=10.10.2.0
+ FLINK_VERSION=1.13.6
+ SPARK_VERSION=2.4.8
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1136hive239spark248
+elif [[ ${SPARK_RUNTIME} == 'spark3.0.2' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=3.1.3
+ DERBY_VERSION=10.14.1.0
+ FLINK_VERSION=1.14.6
+ SPARK_VERSION=3.0.2
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1146hive313spark302
+elif [[ ${SPARK_RUNTIME} == 'spark3.1.3' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=3.1.3
+ DERBY_VERSION=10.14.1.0
+ FLINK_VERSION=1.13.6
+ SPARK_VERSION=3.1.3
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1136hive313spark313
+elif [[ ${SPARK_RUNTIME} == 'spark3.2.3' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=3.1.3
+ DERBY_VERSION=10.14.1.0
+ FLINK_VERSION=1.14.6
+ SPARK_VERSION=3.2.3
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1146hive313spark323


nit: older Spark versions can be removed.

yihua · 2023-07-25T19:01:59Z

 <flink.connector.kafka.artifactId>flink-connector-kafka</flink.connector.kafka.artifactId>
 <flink.hadoop.compatibility.artifactId>flink-hadoop-compatibility_2.12</flink.hadoop.compatibility.artifactId>
- <rocksdbjni.version>5.17.2</rocksdbjni.version>
+ <rocksdbjni.version>7.5.3</rocksdbjni.version>


After reading the code, RocksDbBasedFileSystemView is the only place using RocksDb. So this should be ok.

yihua · 2023-07-25T19:05:31Z

CI has passed for 6b33d37. No need to rerun CI again.

CTTY changed the title ~~[DNM] Skip tests for Java 17~~[HUDI-6509] Skip tests for Java 17Jul 10, 2023

CTTY changed the title ~~[HUDI-6509] Skip tests for Java 17~~[HUDI-6509] Add GitHub CI for Java 17Jul 10, 2023

CTTY force-pushed the ctty/master-skiptest-j17 branch from cb10175 to 8ea8e3aCompare July 12, 2023 04:41

yihua self-assigned this Jul 13, 2023

CTTY changed the title ~~[HUDI-6509] Add GitHub CI for Java 17~~[DNM] [HUDI-6509] Add GitHub CI for Java 17Jul 15, 2023

CTTY changed the title ~~[DNM] [HUDI-6509] Add GitHub CI for Java 17~~[HUDI-6509] Add GitHub CI for Java 17Jul 16, 2023

CTTY mentioned this pull request Jul 16, 2023
[DNM] Testing Java 17 #8949
Closed
4 tasks

CTTY force-pushed the ctty/master-skiptest-j17 branch from a038fd6 to c79f37cCompare July 16, 2023 06:12

Add Hadoop conf to HiveConf for HiveSyncConfig
68f6b69

yihua added dependencies Dependency updates priority:blocker Production down; release blocker release-0.14.0 labels Jul 18, 2023

yihua reviewed Jul 18, 2023
View reviewed changes

CTTY added 14 commits July 23, 2023 23:29

Add GitHub CI for spark33/34 Java 17
10f012b

Testing docker java 17 test
b8035db

minor
e083529

test
7a86dee

All docker java17 tests passed, adding all other CIs back
6bdcdac

minor
5a07bf7

trigger github ci
e8cd0e2

fix java17 test flag
8d3cca1

Rebase onto master. Migrate one new test
44d10bd

stablize TestHoodieWrapperFileSystem
100b397

trigger ci
60eb622

add ps in alpine image
8d76f7d

minor
3f8c949

minor
26777da

CTTY added 17 commits July 23, 2023 23:29

fix path
de24436

fixing ssh command
323c1de

only test for spark 3.4
6a86262

trigger ci again
8e54816

Have 6 datanodes
5a0ed39

10 datanodes
10f4cb2

have 8 datanodes
cdaff5f

5 nodes
409e7d8

3 nodes
172b237

run java 17 ci first
d018c71

adjust ci
9bfcf7c

minor
150f0be

run clean test
d496aed

seperate CIs
91a5dec

Fix core site
c8540ca

Fix core site
3723d50

fix core-site
9bc5072

CTTY force-pushed the ctty/master-skiptest-j17 branch from 95b9ca4 to 9bc5072Compare July 24, 2023 06:29

CTTY added 2 commits July 24, 2023 11:05

try removing spark-streaming test dep
91c5a05

add spark streaming back
6b33d37

Revert "Add Hadoop conf to HiveConf for HiveSyncConfig"
8b62b65
This reverts commit 68f6b69.

yihua approved these changes Jul 25, 2023
View reviewed changes

yihua merged commit 03bc554 into apache:masterJul 25, 2023

CTTY deleted the ctty/master-skiptest-j17 branch July 25, 2023 19:15

jonvex mentioned this pull request Nov 7, 2023
[MINOR] Remove rocksdb version from m1 profile #10006
Merged
4 tasks

mzheng-plaid mentioned this pull request Nov 5, 2024
[SUPPORT] Current state of Java 17 support #12207
Closed

hudi-bot mentioned this pull request Dec 9, 2025
Java 17 runtime support #17236
Closed

rangareddy mentioned this pull request Dec 9, 2025
Remove unnecessary usage of rocksdb usage in CI tests #17249
Closed

Uh oh!

Conversation

CTTY commented Jul 6, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

Uh oh!

CTTY commented Jul 12, 2023

Uh oh!

CTTY commented Jul 18, 2023

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTYJul 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hudi-bot commented Jul 24, 2023

CI report:

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua commented Jul 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CTTY commented Jul 6, 2023•
edited
Loading

CTTYJul 18, 2023•
edited
Loading