diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala index e21ff7b8f7913..a72079274f7dd 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala @@ -88,7 +88,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S /** Default properties present in the currently defined defaults file. */ lazy val defaultSparkProperties: HashMap[String, String] = { val defaultProperties = new HashMap[String, String]() - if (verbose) { + if (verbose && propertiesFile != null) { logInfo(log"Using properties file: ${MDC(PATH, propertiesFile)}") } Option(propertiesFile).foreach { filename => diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md index 76ccfc66746ef..80bc25fd62a74 100644 --- a/docs/core-migration-guide.md +++ b/docs/core-migration-guide.md @@ -50,7 +50,7 @@ license: | - Since Spark 4.0, Spark performs speculative executions less aggressively with `spark.speculation.multiplier=3` and `spark.speculation.quantile=0.9`. To restore the legacy behavior, you can set `spark.speculation.multiplier=1.5` and `spark.speculation.quantile=0.75`. -- Since Spark 4.0, when configurations are provided to `spark-submit` or `spark-shell` via `--properties-file` parameter, Spark will also load extra configurations from `SPARK_CONF_DIR/spark-defaults.conf`. Those configurations defined in the former take higher precedence over those from the latter. +- Since Spark 4.0, when configurations are provided to `spark-submit` or `spark-shell` via `--properties-file` parameter, Spark will also load extra configurations from `SPARK_CONF_DIR/spark-defaults.conf`. Those configurations defined in the former take higher precedence over those from the latter. Other components such as Spark thrift server, connect server, SQL shell that call `spark-submit` internally will also have the same behavior change. ## Upgrading from Core 3.4 to 3.5