Swappable modules classloader — restartless add/upgrade/remove of dependency JARs

[ThuF]

Part 2 of the Dynamic dependency management epic (#6776). Depends on part 1 — #6777 (resolver + descriptor).
Delivers the headline capability: a change to any project's maven dependencies takes effect
without restarting the platform — new AOT module JARs register their classes and payload, new
third-party libraries become importable from registry .java sources and JS, and removed/upgraded
JARs leave cleanly.

Why this is a small change, not a big one

The engine already has every structural piece except one:

• JavaLoader (components/engine/engine-java/.../runtime/JavaLoader.java) maintains the
  installed class set as a generation — a union of the registry-compiled sub-generation and the
  AOT compiledGeneration — and applyGeneration(...) is documented source-agnostic;
  installCompiledModules(...) is documented idempotent ("re-invoking replaces the compiled set").
• ClientClassLoaderHolder (components/core/core-java/.../runtime/ClientClassLoaderHolder.java)
  is the single swap-point already consumed by Flowable
  (components/engine/engine-bpm-flowable/.../config/ClientAwareClassLoader.java,
  FlowableClientClassLoaderRefresher), Camel
  (components/engine/engine-camel/.../invoke/DirigibleJavaInvokerImpl.java) and the Beans API
  (components/api/api-modules-java/.../component/Beans.java).
• Old-generation lifecycle (drain in-flight, Metaspace reclaimed at GC) is the established contract.

The one hardwired assumption: classes come from the application classloader, because /modules
is on the launch classpath. This issue replaces that assumption with a swappable layer.

Design

New: ModulesClassLoader + holder

In core-java (next to ClientClassLoader):

• ModulesClassLoader extends URLClassLoader — over the resolved JAR set (absolute paths inside
  the Maven local repository — immutable versioned paths, never a mutable copy dir; never
  overwrite a JAR an open loader may hold). Parent = the application classloader
  (JavaHandler.class.getClassLoader() as anchored today). Parent-first delegation — no
  child-first tricks.
• ModulesClassLoaderHolder — mirrors ClientClassLoaderHolder: current(), swap(next).
  Initial generation is built at startup from (a) JARs already on loader.path (compat: everything
  in /modules keeps working exactly as before — for these, classes load from the app CL parent
  anyway) and (b) the resolver's current union result.

The swap pipeline

A new DependencySynchronizer (in core-dependencies, or wired into the existing synchronizer
framework — follow the artefact-synchronizer conventions in components/core/core-initializers)
watches project.json changes, and:

1. Runs union resolution (part 1) → new JAR set.
2. Builds ModulesClassLoader generation N+1 over the new set.
3. Registry payload: for each added/changed JAR, lays META-INF/dirigible/<project>/** into
   registry/public/<project>/ ; for each removed JAR, removes that project prefix. This requires
   refactoring ClasspathExpander
   (components/core/core-initializers/.../classpath/ClasspathExpander.java) to expose per-JAR
   expand(Path jar) / remove(String project) alongside the existing startup sweep (keep the
   .skip marker behavior).
4. AOT classes: re-runs compiled-module discovery against the new loader and calls
   JavaLoader.installCompiledModules(...). This requires CompiledModuleClassProvider
   (components/engine/engine-java/.../runtime/CompiledModuleClassProvider.java) to:
   • accept the classloader to scan/load through (today it uses getClass().getClassLoader()),
   • expose a public rediscover(ClassLoader) in addition to the ApplicationReadyEvent hook.
5. Compile classpath: invalidates ClassPathIndex
   (components/engine/engine-java/.../runtime/ClassPathIndex.java) — today it caches once in an
   AtomicReference and only knows BOOT-INF/lib + loader.path; it must gain an
   invalidate()/listener and include the resolved JAR set — then triggers a registry .java
   rebuild so client sources can compile against the new dependencies.
6. Swaps ModulesClassLoaderHolder; the generation parent used in JavaLoader.rebuild(...)
   (currently JavaHandler.class.getClassLoader(), see the ClientClassLoader construction)
   becomes ModulesClassLoaderHolder.current().
7. Emits an application event (DependenciesChangedEvent) with added/removed/mediated coordinates,
   for the monitoring perspective and downstream listeners.

Consumers that must resolve through the holder chain

Audit and fix lookups that today assume the application classloader can see everything:

• GraalJS host access (Java.type(...) in the JS engine) must resolve through
  ClientClassLoaderHolder.current() (whose parent chain now includes the modules loader).
• Flowable/Camel already go through the holder — verify with tests, don't assume.

Explicit non-behaviors (state them in javadoc and docs)

• No Spring bean scanning / auto-configuration from module JARs (classes flow through
  ComponentContainer / JavaClassConsumers, as today).
• Static state in module classes re-initializes on upgrade (same contract as client classes).
• JARs containing native libraries (.so/.dylib/.dll entries) are rejected on the module
  tier with an error naming the file and pointing at scope: "platform" (phase 3) — the JVM
  allows a native lib in only one classloader, so a swappable loader would break on first upgrade.
• Parent-first shadowing: a dependency also present in the platform's BOOT-INF/lib resolves to
  the platform's version. Detection/reporting is phase 4; this phase logs a WARN when a resolved
  artifact's groupId:artifactId is known to be on the platform classpath.

Implementation checklist

• ModulesClassLoader + ModulesClassLoaderHolder in core-java, with javadoc mirroring the
  quality of ClientClassLoader's (explain the generation lifecycle and the immutable-path rule).
• ClasspathExpander: extract per-JAR expand(Path) / remove(String project); startup sweep
  delegates to them.
• CompiledModuleClassProvider: parameterize the classloader; add rediscover(ClassLoader).
• ClassPathIndex: invalidate() + include resolved JARs.
• JavaLoader.rebuild(...): generation parent from ModulesClassLoaderHolder.
• DependencySynchronizer implementing the seven-step pipeline; concurrency: one swap at a
  time, serialized with JavaLoader.rebuild (both already synchronized on JavaLoader — keep
  that lock discipline).
• JS engine host-access lookup through the holder chain.
• Monitoring: expose current generation number and live/pinned old-generation count.

Tests (part of the PR)

Unit:

• ModulesClassLoaderTest — parent-first delegation; class visible from child
  ClientClassLoader generations; two generations over different JAR sets are independent.
• ClasspathExpanderTest — per-JAR expand/remove round-trip, .skip honored.
• CompiledModuleClassProviderTest — extend the existing test
  (components/engine/engine-java/src/test/java/.../CompiledModuleClassProviderTest.java) with a
  marker discovered through a custom classloader (fixture JAR built by the test).

Integration (tests/tests-integrations/.../api/DynamicDependenciesIT.java, extending
IntegrationTest; all repositories are file:// fixtures — no network):

1. Add a module restartlessly: boot with no deps → publish a fixture AOT module JAR (containing
   one @Controller class listed in .compiled + a payload file) into the fixture repo → add the
   maven entry to the project's project.json → await (Awaitility, as used across the ITs) the
   controller answering over HTTP and the payload present in the registry. Assert zero runtime
   javac for the AOT classes.
2. Third-party lib for client code: add a fixture plain-JAR dependency; a registry .java
   source importing it compiles and serves a request using it.
3. Upgrade: bump the fixture module 1.0.0 → 1.1.0 (changed response body) in project.json;
   await the new response; assert the old class no longer served and payload replaced.
4. Remove: drop the entry; await 404 from the controller and payload gone from the registry.
5. Native-lib rejection: a fixture JAR containing a dummy lib/foo.so is rejected with the
   pointed error; platform keeps running.

The fixture AOT JARs should be produced by a small test utility in tests-framework (build
.class files with the JDK compiler API + write .compiled + payload) so the examples are
readable in the test source — these tests double as the mechanism's executable documentation.

PR requirements

• PR description: worked example — project.json diff, the log lines of one full swap
  (resolve → expand → rediscover → rebuild → swap), before/after HTTP responses of test Fix SchedulerServlet shutdown procedure. #3.
• Code comments/javadoc: ModulesClassLoader javadoc carries the generation-lifecycle example;
  DependencySynchronizer javadoc documents the seven-step pipeline.
• EPL-2.0 headers, dirigible-formatter.xml.

Acceptance criteria

• The five ITs above green, no network.
• /modules boot behavior byte-for-byte compatible when no maven deps are declared.
• A failed resolution or a bad JAR mid-swap leaves generation N installed and serving (no partial
  swap) — add an IT for a JAR that fails to expand.
• Heap-dump observability: at most the expected number of ModulesClassLoader instances alive
  after ITs (no loader leak introduced by the pipeline itself).