[SPARK-45147][CORE] Remove System.setSecurityManager usage

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to remove the deprecate Java System.setSecurityManager usage for Apache Spark 4.0.

Note that this is the only usage in Apache Spark AS-IS code.

$ git grep setSecurityManager
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala: System.setSecurityManager(sm)
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala: System.setSecurityManager(currentSm)

This usage was added at Apache Spark 1.5.0.

• [SPARK-6324] [core] Centralize handling of script usage messages. #5841

Since Apache Spark 2.4.0, we don't need setSecurityManager due to the following improvement.

• [SPARK-22941][core] Do not exit JVM when submit fails with in-process launcher. #20925

Why are the changes needed?

$ java -version
openjdk version "21-ea" 2023-09-19
OpenJDK Runtime Environment (build 21-ea+32-2482)
OpenJDK 64-Bit Server VM (build 21-ea+32-2482, mixed mode, sharing)

max spark-3.5.0-bin-hadoop3:$ bin/spark-sql --help
...
CLI options:
Exception in thread "main" java.lang.UnsupportedOperationException:
The Security Manager is deprecated and will be removed in a future release
at java.base/java.lang.System.setSecurityManager(System.java:429)
at org.apache.spark.deploy.SparkSubmitArguments.getSqlShellOptions(SparkSubmitArguments.scala:623)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual test.

$ build/sbt test:package -Phive -Phive-thriftserver
$ bin/spark-sql --help
...
CLI options:
-d,--define <key=value> Variable substitution to apply to Hive
commands. e.g. -d A=B or --define A=B
--database <databasename> Specify the database to use
-e <quoted-query-string> SQL from command line
-f <filename> SQL from files
-H,--help Print help information
--hiveconf <property=value> Use value for given property
--hivevar <key=value> Variable substitution to apply to Hive
commands. e.g. --hivevar A=B
-i <filename> Initialization SQL file
-S,--silent Silent mode in interactive shell
-v,--verbose Verbose mode (echo executed SQL to the
console)

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this PR when you have some time, @LuciferYang ?

[LuciferYang]

LGTM

[dongjoon-hyun]

Thank you, @LuciferYang .

[dongjoon-hyun]

Thank you, @yaooqinn .

[dongjoon-hyun]

core and launcher module tests passed.

[dongjoon-hyun]

Merged to master.