HDDS-11450. Avoid managing versions for transitive-only dependencies

[adoroszlai]

What changes were proposed in this pull request?

Ozone should not override version of transitive dependencies which are not used directly. E.g. re2j via Hadoop, is only used by Hadoop. Bumping its version in Ozone may cause incompatibilities at runtime.

Currently:

• When building each Ozone component, its dependencies are listed in a file, to be used at runtime as classpath.
• Dependencies are copied to the share/ozone/lib directory when building the ozone-dist module. It depends on all other Ozone modules directly or transitively. Since ozone-dist is a single module, third-party transitive dependency versions are resolved by Maven to a single version. Only that version is copied to share/ozone/lib. However, various Ozone components may depend on different versions of the same third-party module. These different versions can end up in the classpath files, but will not be found at runtime.
• To fix the problem of missing jars due to different versions, Ozone started to manage versions of third-party dependencies (HDDS-7372, but even before that to some degree). Now dependency versions may be changed independently, and version of transitive dependency may be too old or too new for the direct dependency.
  • New version of some dependency bumps its own dependencies, but we keep using older versions of the transitive dependencies.
  • We bump transitive dependency version without bumping the direct dependency that requires it.
    Both of these could result in incompatibilities, which may go unnoticed in CI.

This PR proposes to:

• Allow multiple versions of the same dependency by copying them when building each module. dist-layout-stitching copies dependencies from per-module directories to the final location when building ozone-dist.
• Remove some transitive-only dependencies. Some exclusions are necessary to resolve dependency convergence errors.
• Keep dependencies upgraded due to security issues.

Dependency changes:

@@ -3,6 +3,7 @@
annotations-4.1.1.4.jar
aopalliance-1.0.jar
aopalliance-repackaged-2.6.1.jar
+asm-9.1.jar
asm-9.2.jar
asm-analysis-9.2.jar
asm-commons-9.2.jar
@@ -39,7 +40,7 @@
curator-framework-4.2.0.jar
derby-10.14.2.0.jar
disruptor-3.4.4.jar
-dnsjava-2.1.9.jar
+dnsjava-2.1.7.jar
error_prone_annotations-2.29.2.jar
failureaccess-1.0.1.jar
gethostname4j-0.0.2.jar
@@ -62,7 +63,7 @@
hadoop-common-3.3.6.jar
hadoop-hdfs-3.3.6.jar
hadoop-hdfs-client-3.3.6.jar
-hadoop-shaded-guava-1.2.0.jar
+hadoop-shaded-guava-1.1.1.jar
hadoop-shaded-protobuf_3_7-1.1.1.jar
hdds-annotation-processing-1.5.0-SNAPSHOT.jar
hdds-client-1.5.0-SNAPSHOT.jar
@@ -158,7 +159,7 @@
jnr-ffi-2.2.16.jar
jnr-posix-3.1.19.jar
jnr-x86asm-1.0.2.jar
-joda-time-2.12.7.jar
+joda-time-2.10.6.jar
jooq-3.11.10.jar
jooq-codegen-3.11.10.jar
jooq-meta-3.11.10.jar
@@ -254,7 +255,7 @@
ratis-server-api-3.1.0.jar
ratis-thirdparty-misc-1.0.6.jar
ratis-tools-3.1.0.jar
-re2j-1.7.jar
+re2j-1.1.jar
reflections-0.10.2.jar
reload4j-1.2.25.jar
rocksdb-checkpoint-differ-1.5.0-SNAPSHOT.jar
@@ -271,11 +272,12 @@
spring-jdbc-5.3.39.jar
spring-tx-5.3.39.jar
sqlite-jdbc-3.46.1.0.jar
-stax2-api-4.2.2.jar
+stax2-api-4.2.1.jar
txw2-2.3.9.jar
vault-java-driver-5.1.0.jar
weld-servlet-shaded-3.1.9.Final.jar
woodstox-core-5.4.0.jar
+woodstox-core-6.5.1.jar
zookeeper-3.8.4.jar
zookeeper-jute-3.8.4.jar
zstd-jni-1.5.6-4.jar

https://issues.apache.org/jira/browse/HDDS-11450

How was this patch tested?

CI:
https://github.com/adoroszlai/ozone/actions/runs/10825596616

[adoroszlai]

Additionally, we could also get rid of the dependency convergence check to avoid the need for exclusions (see adoroszlai@bba79f2).

[adoroszlai]

Extracted part of this to #7325.

[adoroszlai]

I plan to create separate PR for each dependency for easier review.