[SPARK-45197][CORE] Make StandaloneRestServer add JavaModuleOptions to drivers

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to make StandaloneRestServer add JavaModuleOptions to drivers by default.

Why are the changes needed?

Since Apache Spark 3.3.0 (SPARK-36796, #34153), SparkContext adds JavaModuleOptions by default.

We had better add JavaModuleOptions when StandaloneRestServer receives submissions via REST API, too. Otherwise, it fails like the following if the users don't set it manually.

SUBMISSION

$ SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true" sbin/start-master.sh
$ curl -s -k -XPOST http://yourserver:6066/v1/submissions/create \
    --header "Content-Type:application/json;charset=UTF-8" \
    --data '{
      "appResource": "",
      "sparkProperties": {
        "spark.master": "local[2]",
        "spark.app.name": "Test 1",
        "spark.submit.deployMode": "cluster",
        "spark.jars": "/Users/dongjoon/APACHE/spark-release/spark-3.5.0-bin-hadoop3/examples/jars/spark-examples_2.12-3.5.0.jar"
      },
      "clientSparkVersion": "",
      "mainClass": "org.apache.spark.examples.SparkPi",
      "environmentVariables": {},
      "action": "CreateSubmissionRequest",
      "appArgs": []
    }'

DRIVER stderr LOG

Exception in thread "main" java.lang.reflect.InvocationTargetException
...
at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$
(in unnamed module @0x6d7a93c9) cannot access class sun.nio.ch.DirectBuffer
(in module java.base) because module java.base does not export sun.nio.ch
to unnamed module @0x6d7a93c9

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs with the newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

This private method arguments are revised for better testing by taking advantage of dependency injection pattern.

[dongjoon-hyun]

Thank you always, @yaooqinn and @LuciferYang !

[dongjoon-hyun]

Merged to master.