Skip to content

Commit

Permalink
[SPARK-39361] Don't use Log4J2's extended throwable conversion patter…
Browse files Browse the repository at this point in the history
…n in default logging configurations

### What changes were proposed in this pull request?

This PR addresses a performance problem in Log4J 2 related to exception logging: in certain scenarios I observed that Log4J2's default exception stacktrace logging can be ~10x slower than Log4J 1.

The problem stems from a new log pattern format in Log4J2 called ["extended exception"](https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternExtendedException), which enriches the regular stacktrace string with information on the name of the JAR files that contained the classes in each stack frame.

Log4J queries the classloader to determine the source JAR for each class. This isn't cheap, but this information is cached and reused in future exception logging calls. In certain scenarios involving runtime-generated classes, this lookup will fail and the failed lookup result will _not_ be cached. As a result, expensive classloading operations will be performed every time such an exception is logged. In addition to being very slow, these operations take out a lock on the classloader and thus can cause severe lock contention if multiple threads are logging errors. This issue is described in more detail in [a comment on a Log4J2 JIRA](https://issues.apache.org/jira/browse/LOG4J2-2391?focusedCommentId=16667140&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16667140) and in a linked blogpost. Spark frequently uses generated classes and lambdas and thus Spark executor logs will almost always trigger this edge-case and suffer from poor performance.

By default, if you do not specify an explicit exception format in your logging pattern then Log4J2 will add this "extended exception" pattern (see PatternLayout's alwaysWriteExceptions flag in Log4J's documentation, plus [the code implementing that flag](https://github.com/apache/logging-log4j2/blob/d6c8ab0863c551cdf0f8a5b1966ab45e3cddf572/log4j-core/src/main/java/org/apache/logging/log4j/core/pattern/PatternParser.java#L206-L209) in Log4J2).

In this PR, I have updated Spark's default Log4J2 configurations so that each pattern layout includes an explicit %ex so that it uses the normal (non-extended) exception logging format. This is the workaround that is currently recommended on the Log4J JIRA.

### Why are the changes needed?

Avoid performance regressions in Spark programs which use Spark's default Log4J 2 configuration and log many exceptions. Although it's true that any program logging exceptions at a high rate should probably just fix the source of the exceptions, I think it's still a good idea for us to try to fix this out-of-the-box performance difference so that users' existing workloads do not regress when upgrading to 3.3.0.

### Does this PR introduce _any_ user-facing change?

Yes: it changes the default exception logging format so that it matches Log4J 1's default rather than Log4J 2's. The new format is consistent with behavior in previous Spark versions, but is different than the behavior in the current Spark 3.3.0-rc3.

### How was this patch tested?

Existing tests.

Closes apache#36747 from JoshRosen/disable-log4j2-extended-exception-pattern.

Authored-by: Josh Rosen <joshrosen@databricks.com>
Signed-off-by: Josh Rosen <joshrosen@databricks.com>
JoshRosen committed Jun 2, 2022
1 parent a760975 commit fd45c36
Showing 38 changed files with 53 additions and 47 deletions.
2 changes: 1 addition & 1 deletion R/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.eclipse.jetty
2 changes: 1 addition & 1 deletion common/kvstore/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Silence verbose logs from 3rd-party libraries.
logger.netty.name = io.netty
2 changes: 1 addition & 1 deletion common/network-common/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Silence verbose logs from 3rd-party libraries.
logger.netty.name = io.netty
Original file line number Diff line number Diff line change
@@ -24,4 +24,4 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex
8 changes: 7 additions & 1 deletion conf/log4j2.properties.template
Original file line number Diff line number Diff line change
@@ -19,11 +19,17 @@
rootLogger.level = info
rootLogger.appenderRef.stdout.ref = console

# In the pattern layout configuration below, we specify an explicit `%ex` conversion
# pattern for logging Throwables. If this was omitted, then (by default) Log4J would
# implicitly add an `%xEx` conversion pattern which logs stacktraces with additional
# class packaging information. That extra information can sometimes add a substantial
# performance overhead, so we disable it in our default logging config.
# For more information, see SPARK-39361.
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

# Set the default spark-shell/spark-sql log level to WARN. When running the
# spark-shell/spark-sql, the log level for these classes is used to overwrite
2 changes: 1 addition & 1 deletion connector/avro/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -33,7 +33,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
2 changes: 1 addition & 1 deletion connector/kafka-0-10/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark-project.jetty
4 changes: 2 additions & 2 deletions connector/kinesis-asl/src/main/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -23,14 +23,14 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %p %c{1}: %m%n%ex

# Console appender
appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

# Settings to quiet third party logs that are too verbose
logger.jetty1.name = org.sparkproject.jetty
2 changes: 1 addition & 1 deletion connector/kinesis-asl/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

# Settings to quiet third party logs that are too verbose
logger.jetty.name = org.sparkproject.jetty
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/TestUtils.scala
Original file line number Diff line number Diff line change
@@ -431,7 +431,7 @@ private[spark] object TestUtils {
val appenderBuilder = builder.newAppender("console", "CONSOLE")
.addAttribute("target", ConsoleAppender.Target.SYSTEM_ERR)
appenderBuilder.add(builder.newLayout("PatternLayout")
.addAttribute("pattern", "%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n"))
.addAttribute("pattern", "%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex"))
builder.add(appenderBuilder)
builder.add(builder.newRootLogger(level).add(builder.newAppenderRef("console")))
val configuration = builder.build()
Original file line number Diff line number Diff line change
@@ -42,7 +42,7 @@ private[spark] class DriverLogger(conf: SparkConf) extends Logging {

private val UPLOAD_CHUNK_SIZE = 1024 * 1024
private val UPLOAD_INTERVAL_IN_SECS = 5
private val DEFAULT_LAYOUT = "%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n"
private val DEFAULT_LAYOUT = "%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex"
private val LOG_FILE_PERMISSIONS = new FsPermission(Integer.parseInt("770", 8).toShort)

private val localLogFile: String = FileUtils.getFile(
4 changes: 2 additions & 2 deletions core/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -32,7 +32,7 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %t: %m%n
appender.console.layout.pattern = %t: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
2 changes: 1 addition & 1 deletion docs/configuration.md
Original file line number Diff line number Diff line change
@@ -449,7 +449,7 @@ of the most common options to set are:
</tr>
<tr>
<td><code>spark.driver.log.layout</code></td>
<td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n</td>
<td>%d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex</td>
<td>
The layout for the driver logs that are synced to <code>spark.driver.log.dfsDir</code>. If this is not configured,
it uses the layout for the first appender defined in log4j2.properties. If that is also not configured, driver logs
2 changes: 1 addition & 1 deletion graphx/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
4 changes: 2 additions & 2 deletions hadoop-cloud/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Tests that launch java subprocesses can set the "test.appender" system property to
# "console" to avoid having the child process's logs overwrite the unit test's
@@ -33,7 +33,7 @@ appender.console.type = Console
appender.console.name = STDERR
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %t: %m%n
appender.console.layout.pattern = %t: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.spark_project.jetty
4 changes: 2 additions & 2 deletions launcher/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -23,13 +23,13 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex

appender.childproc.type = Console
appender.childproc.name = childproc
appender.childproc.target = SYSTEM_ERR
appender.childproc.layout.type = PatternLayout
appender.childproc.layout.pattern = %t: %m%n
appender.childproc.layout.pattern = %t: %m%n%ex

appender.outputredirtest.type = LogAppender
appender.outputredirtest.name = outputredirtest
2 changes: 1 addition & 1 deletion mllib/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
2 changes: 1 addition & 1 deletion repl/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = file
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
2 changes: 1 addition & 1 deletion repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala
Original file line number Diff line number Diff line change
@@ -285,7 +285,7 @@ class ReplSuite extends SparkFunSuite with BeforeAndAfterAll {
|appender.console.target = SYSTEM_ERR
|appender.console.follow = true
|appender.console.layout.type = PatternLayout
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|
|# Set the log level for this class to ERROR same as the default setting.
|logger.repl.name = org.apache.spark.repl.Main
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
Original file line number Diff line number Diff line change
@@ -23,4 +23,4 @@ appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n
appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n%ex
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/integration-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ private[spark] trait DecommissionSuite { k8sSuite: KubernetesSuite =>
|appender.console.name = console
|appender.console.target = SYSTEM_OUT
|appender.console.layout.type = PatternLayout
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|
|logger.spark.name = org.apache.spark
|logger.spark.level = debug
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from a few verbose libraries.
logger.jersey.name = com.sun.jersey
Original file line number Diff line number Diff line change
@@ -53,7 +53,7 @@ abstract class BaseYarnClusterSuite
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
|logger.jetty.name = org.sparkproject.jetty
|logger.jetty.level = warn
|logger.eclipse.name = org.eclipse.jetty
2 changes: 1 addition & 1 deletion sql/catalyst/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty
4 changes: 2 additions & 2 deletions sql/core/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn

@@ -34,7 +34,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Set the logger level of File Appender to WARN
appender.file.filter.threshold.type = ThresholdFilter
Original file line number Diff line number Diff line change
@@ -29,9 +29,9 @@ public class CLIServiceUtils {

private static final char SEARCH_STRING_ESCAPE = '\\';
public static final StringLayout verboseLayout = PatternLayout.newBuilder().withPattern(
"%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n").build();
"%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n%ex").build();
public static final StringLayout nonVerboseLayout = PatternLayout.newBuilder().withPattern(
"%-5p : %m%n").build();
"%-5p : %m%n%ex").build();

/**
* Convert a SQL search pattern into an equivalent Java Regex.
4 changes: 2 additions & 2 deletions sql/hive-thriftserver/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex

appender.console.filter.1.type = Filters

@@ -43,7 +43,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

appender.file.filter.1.type = Filters

Original file line number Diff line number Diff line change
@@ -1228,7 +1228,7 @@ abstract class HiveThriftServer2TestBase extends SparkFunSuite with BeforeAndAft
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
Original file line number Diff line number Diff line change
@@ -80,7 +80,7 @@ class UISeleniumSuite
|appender.console.name = console
|appender.console.target = SYSTEM_ERR
|appender.console.layout.type = PatternLayout
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
|appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex
""".stripMargin,
new File(s"$tempLog4jConf/log4j2.properties"),
StandardCharsets.UTF_8)
4 changes: 2 additions & 2 deletions sql/hive/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@ appender.console.type = Console
appender.console.name = STDOUT
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n
appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = warn

@@ -34,7 +34,7 @@ appender.file.type = File
appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Set the logger level of File Appender to WARN
appender.file.filter.threshold.type = ThresholdFilter
2 changes: 1 addition & 1 deletion streaming/src/test/resources/log4j2.properties
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ appender.file.name = File
appender.file.fileName = target/unit-tests.log
appender.file.append = true
appender.file.layout.type = PatternLayout
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n
appender.file.layout.pattern = %d{yy/MM/dd HH:mm:ss.SSS} %t %p %c{1}: %m%n%ex

# Ignore messages below warning level from Jetty, because it's a bit verbose
logger.jetty.name = org.sparkproject.jetty

0 comments on commit fd45c36

Please sign in to comment.