[SPARK-45426][CORE] Add support for a ReloadingX509TrustManager

[hasnain-db]

What changes were proposed in this pull request?

This adds in support for trust store reloading, mirroring the Hadoop implementation (see source comments for a link). I believe reusing the existing code instead of adding a dependency is fine license wise (see https://github.com/apache/spark/pull/42685/files#r1333667328)

Why are the changes needed?

This helps us refresh trust stores without needing downtime

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added unit tests (also copied from upstream)

build/sbt
> project network-common
> testOnly org.apache.spark.network.ssl.ReloadingX509TrustManagerSuite

The rest of the changes and integration were tested as part of #42685

Was this patch authored or co-authored using generative AI tooling?

No

[mridulm]

sleep can be spuriously woken up - which can make this test flakey.
Ensure it has actually slept for the requested time (See SystemClock.waitTillTime for an example).

[mridulm]

Instead of relying on timing, and potentially becoming flakey (in case reloader was delayed), I would suggest to add a counter to track the number of times reload happened - and wait on that to change.

This also minimizes the arbitrary sleep values in various tests.

[mridulm]

Comments on this test should apply to others below as well.

[hasnain-db]

@mridulm thanks for the review here. I copied this code as is from Hadoop with no modifications, with the idea that that's well tested and used and we can avoid any risks from changing implementations. Happy to change the implementation here (and potentially upstream), but wanted to double check that that's what you prefer instead of keeping compatibility with a tested implementation. Please let me know.

[mridulm]

If this is to be part of Apache Spark codebase, divergence is to be expected as code evolves and we add functionality relevant to Spark.

[hasnain-db]

Thanks! Will address the comments and take a more detailed look in that case.

[mridulm]

Just couple of minor comments, we can merge once addressed.

[mridulm]

The test failure does not look relevant, but can you retrigger tests please ?

[hasnain-db]

@mridulm done - I can re-request review once tests pass (might take a try or 2)

[mridulm]

The test failure is unrelated.
Merged to master.
Thanks for working on this @hasnain-db !