Usage

Add the log4j2-logstash-layout dependency to your POM file

<dependency>
    <groupId>com.vlkan.log4j2</groupId>
    <artifactId>log4j2-logstash-layout</artifactId>
    <version>${log4j2-logstash-layout.version}</version>
</dependency>

together with a valid log4j-core dependency:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>${log4j2.version}</version>
</dependency>

Below you can find a sample log4j2.xml snippet employing LogstashLayout.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn">
    <Appenders>
        <Console name="CONSOLE" target="SYSTEM_OUT">
            <LogstashLayout dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSSZZZ"
                            eventTemplateUri="classpath:LogstashJsonEventLayoutV1.json"
                            prettyPrintEnabled="true"
                            stackTraceEnabled="true"/>
        </Console>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="CONSOLE"/>
        </Root>
    </Loggers>
</Configuration>

Or using the log4j2.properties file instead:

status = warn

appender.console.name = CONSOLE
appender.console.type = CONSOLE
appender.console.target = SYSTEM_OUT

appender.console.logstash.type = LogstashLayout
appender.console.logstash.dateTimeFormatPattern = yyyy-MM-dd'T'HH:mm:ss.SSSZZZ
appender.console.logstash.eventTemplateUri = classpath:LogstashJsonEventLayoutV1.json
appender.console.logstash.prettyPrintEnabled = true
appender.console.logstash.stackTraceEnabled = true

rootLogger.level = info
rootLogger.appenderRef.stdout.ref = CONSOLE

This generates an output as follows:

{
  "exception": {
    "exception_class": "java.lang.RuntimeException",
    "exception_message": "test",
    "stacktrace": "java.lang.RuntimeException: test\n\tat com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo.main(LogstashLayoutDemo.java:11)\n"
  },
  "line_number": 12,
  "class": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo",
  "@version": 1,
  "source_host": "varlik",
  "message": "Hello, error!",
  "thread_name": "main",
  "@timestamp": "2017-05-25T19:56:23.370+02:00",
  "level": "ERROR",
  "file": "LogstashLayoutDemo.java",
  "method": "main",
  "logger_name": "com.vlkan.log4j2.logstash.layout.demo.LogstashLayoutDemo"
}

LogstashLayout is configured with the following parameters:

Parameter Name	Type	Description
`prettyPrintEnabled`	boolean	enables pretty-printer (defaults to `false`)
`locationInfoEnabled`	boolean	includes the filename and line number in the output (defaults to `false`)
`stackTraceEnabled`	boolean	includes stack traces (defaults to `false`)
`emptyPropertyExclusionEnabled`	boolean	exclude empty and null properties (defaults to `true`)
`dateTimeFormatPattern`	String	timestamp formatter pattern (defaults to `yyyy-MM-dd'T'HH:mm:ss.SSSZZZ`)
`timeZoneId`	String	time zone id (defaults to `TimeZone.getDefault().getID()`)
`mdcKeyPattern`	String	regex to filter MDC keys (does not apply to direct `mdc:key` access)
`ndcPattern`	String	regex to filter NDC items
`eventTemplate`	String	inline JSON template for rendering `LogEvent`s (has priority over `eventTemplateUri`)
`eventTemplateUri`	String	JSON template for rendering `LogEvent`s (defaults to `classpath:LogstashJsonEventLayoutV1.json`)
`eventTemplateAdditionalFields`¹	KeyValuePair	additional key-value pairs appended to the root of the event template
`stackTraceElementTemplate`	String	inline JSON template for rendering `StackTraceElement`s (has priority over `stackTraceElementTemplateUri`)
`stackTraceElementTemplateUri`	String	JSON template for rendering `StackTraceElement`s (defaults to `classpath:Log4j2StackTraceElementLayout.json`)
`lineSeparator`	String	used to separate log outputs (defaults to `System.lineSeparator()`)
`maxByteCount`	int	used to cap the internal `byte[]` buffer used for serialization (defaults to 512 KiB)
`maxStringLength`²	int	truncate string values longer than the specified limit (defaults to 0)
`maxSerializationContextPoolSize`	int	number of cached serialization contexts, i.e., `JsonGenerator`, `byte[]` of size `maxByteCount`, etc. (defaults to 50)
`maxWriterPoolSize`³	int	number of cached `Writer`s backed by `char[]` of size `maxStringLength` or `maxByteCount` (defaults to 50)
`objectMapperFactoryMethod`	String	custom object mapper factory method (defaults to `com.fasterxml.jackson.databind.ObjectMapper.new`)
`mapMessageFormatterIgnored`	boolean	as a temporary work around for LOG4J2-2703, serialize `MapMessage`s using Jackson rather than `MapMessage#getFormattedMessage()` (defaults to `true`)

¹ One can configure additional event template fields as follows:

<LogstashLayout ...>
    <EventTemplateAdditionalFields>
        <KeyValuePair key="serviceName" value="auth-service"/>
        <KeyValuePair key="containerId" value="6ede3f0ca7d9"/>
    </EventTemplateAdditionalFields>
</LogstashLayout>

² Note that string value truncation via maxStringLength can take place both in object keys and values, and this operation does not leave any trace behind. maxStringLength is intended as a soft protection against bogus input and one should always rely on maxByteCount for a hard limit.

eventTemplateUri denotes the URI pointing to the JSON template that will be used while formatting the LogEvents. By default, LogstashLayout ships LogstashJsonEventLayoutV1.json providing the official Logstash JSONEventLayoutV1.

³ maxWriterPoolSize is only used while serializing stack traces into text, that is, for stackTrace:text directive.

{
  "mdc": "${json:mdc}",
  "ndc": "${json:ndc}",
  "exception": {
    "exception_class": "${json:exception:className}",
    "exception_message": "${json:exception:message}",
    "stacktrace": "${json:exception:stackTrace:text}"
  },
  "line_number": "${json:source:lineNumber}",
  "class": "${json:source:className}",
  "@version": 1,
  "source_host": "${hostName}",
  "message": "${json:message}",
  "thread_name": "${json:thread:name}",
  "@timestamp": "${json:timestamp}",
  "level": "${json:level}",
  "file": "${json:source:fileName}",
  "method": "${json:source:methodName}",
  "logger_name": "${json:logger:name}"
}

Similarly, stackTraceElementUri denotes the URI pointing to the JSON template that will be used while formatting the StackTraceElements. By default, LogstashLayout ships classpath:Log4j2StackTraceElementLayout.json providing an identical stack trace structure produced by Log4j 2 JSONLayout.

{
  "class": "${json:stackTraceElement:className}",
  "method": "${json:stackTraceElement:methodName}",
  "file": "${json:stackTraceElement:fileName}",
  "line": "${json:stackTraceElement:lineNumber}"
}

In case of need, you can create your own templates with a structure tailored to your needs. That is, you can add new fields, remove or rename existing ones, change the structure, etc. Please note that eventTemplateUri parameter only supports file and classpath URI schemes.

Below is the list of allowed LogEvent template variables that will be replaced while rendering the JSON output.

Variable Name	Description
`endOfBatch`	`logEvent.isEndOfBatch()`
`exception:className`	`logEvent.getThrown().getClass().getCanonicalName()`
`exception:message`	`logEvent.getThrown().getMessage()`
`exception:stackTrace`	`logEvent.getThrown().getStackTrace()` (inactive when `stackTraceEnabled=false`)
`exception:stackTrace:text`	`logEvent.getThrown().printStackTrace()` (inactive when `stackTraceEnabled=false`)
`exceptionRootCause:className`	the innermost `exception:className` in causal chain
`exceptionRootCause:message`	the innermost `exception:message` in causal chain
`exceptionRootCause:stackTrace[:text]`	the innermost `exception:stackTrace[:text]` in causal chain
`level`	`logEvent.getLevel()`
`logger:fqcn`	`logEvent.getLoggerFqcn()`
`logger:name`	`logEvent.getLoggerName()`
`main:<key>`	performs Main Argument Lookup for the given `key`
`map:<key>`	performs Map Lookup for the given `key`
`marker:name`	`logEvent.getMarker.getName()`
`mdc`	Mapped Diagnostic Context `Map<String, String>` returned by `logEvent.getContextData()`
`mdc:<key>`	Mapped Diagnostic Context `String` associated with `key` (`mdcKeyPattern` is discarded)
`message`	`logEvent.getFormattedMessage()`
`message:json`	if `logEvent.getMessage()` is of type `MultiformatMessage` and supports JSON, its read value, otherwise, `{"message": <formattedMessage>}` object
`ndc`	Nested Diagnostic Context `String[]` returned by `logEvent.getContextStack()`
`source:className`	`logEvent.getSource().getClassName()`
`source:fileName`	`logEvent.getSource().getFileName()` (inactive when `locationInfoEnabled=false`)
`source:lineNumber`	`logEvent.getSource().getLineNumber()` (inactive when `locationInfoEnabled=false`)
`source:methodName`	`logEvent.getSource().getMethodName()`
`thread:id`	`logEvent.getThreadId()`
`thread:name`	`logEvent.getThreadName()`
`thread:priority`	`logEvent.getThreadPriority()`
`timestamp`	`logEvent.getTimeMillis()` formatted using `dateTimeFormatPattern` and `timeZoneId`
`timestamp:millis`	`logEvent.getTimeMillis()`
`timestamp:nanos`	`logEvent.getNanoTime()`

JSON field lookups are performed using the ${json:<variable-name>} scheme where <variable-name> is defined as <resolver-name>[:<resolver-key>]. Characters following colon (:) are treated as the resolver-key.

Log4j 2 Lookups (e.g., ${java:version}, ${env:USER}, ${date:MM-dd-yyyy}) are supported in templates too. Though note that while ${json:...} template variables are expected to occupy an entire field, that is, "level": "${json:level}", a lookup can be mixed within a regular text: "myCustomField": "Hello, ${env:USER}!".

Similarly, below is the list of allowed StackTraceElement template variables:

Variable Name	Description
`stackTraceElement:className`	`stackTraceElement.getClassName()`
`stackTraceElement:methodName`	`stackTraceElement.getMethodName()`
`stackTraceElement:fileName`	`stackTraceElement.getFileName()`
`stackTraceElement:lineNumber`	`stackTraceElement.getLineNumber()`

As in LogEvent templates, StackTraceElement templates support Log4j 2 lookups too.

See layout-demo directory for a sample application demonstrating the usage of LogstashLayout.

Predefined Templates

log4j2-logstash-layout artifact contains the following predefined templates:

LogstashJsonEventLayoutV1.json described in log4j-jsonevent-layout
EcsLayout.json described by the Elastic Common Schema (ECS) specification

Features

Below is a feature comparison matrix between LogstashLayout and alternatives.

Feature	`LogstashLayout`	`JsonLayout`	`EcsLayout`
Java version	8	7⁴	6
Dependencies	Jackson	Jackson	None
Full schema customization?	✓	✕	✕
Timestamp customization?	✓	✕	✕
(Almost) garbage-free?	✓	✕	✓
Custom typed `Message` serialization?	✓	✕	✓⁵
Custom typed `MDC` value serialization?	✓	✕	✕
Rendering stack traces as array?	✓	✓	✓
Enabling/Disabling JSON pretty print?	✓	✓	✕
Additional fields?	✓	✓	✓

⁴ Log4j 2.4 and greater requires Java 7, versions 2.0-alpha1 to 2.3 required Java 6.

⁵ Only for ObjectMessages and if Jackson is in the classpath.

Fat JAR

Project also contains a log4j2-logstash-layout-fatjar artifact which includes all its transitive dependencies in a separate shaded package (to avoid the JAR Hell) with the exception of log4j-core, that you need to include separately.

This might come handy if you want to use this plugin along with already compiled applications, e.g., Elasticsearch 5.x and 6.x versions, which requires Log4j 2.

Appender Support

log4j2-logstash-layout is all about providing a highly customizable JSON schema for your logs. Though this does not necessarily mean that all of its features are expected to be supported by every appender in the market. For instance, while prettyPrintEnabled=true works fine with log4j2-redis-appender, it should be turned off for Logstash's log4j-json file input type. (See Pretty printing in Logstash issue.) Make sure you configure log4j2-logstash-layout properly in a way that is aligned with your appender of preference.

Performance

The source code contains a benchmark comparing LogstashLayout performance with JsonLayout (shipped by default in Log4j 2) and EcsLayout (shipped by Elastic). There JMH is used to assess the rendering performance of these layouts. In the tests, different LogEvent profiles are employed:

full: LogEvent contains MDC, NDC, and an exception.
lite: LogEvent has no MDC, NDC, or exception attachment.

To give an idea, we ran the benchmark with the following settings:

CPU: Intel i7 2.70GHz (x86-64, confined java process to a single core using taskset -c 0)
JVM: Java HotSpot 1.8.0_192 (-XX:+TieredCompilation, -XX:+AggressiveOpts)
OS: Xubuntu 18.04.3 (4.15.0-54-generic, x86-64)
LogstashLayout4{Ecs,Json}Layout used default settings with the following exceptions:
- emptyPropertyExclusionEnabled: false
- stackTraceEnabled: true
- maxByteCount: 4096
JsonLayout used in two different flavors:
- DefaultJsonLayout: default settings
- CustomJsonLayout: default settings with an additional "@version": 1 field (this forces instantiation of a wrapper class to obtain the necessary Jackson view)
EcsLayout used with the following configurations:
- serviceName: benchmark
- additionalFields: new KeyValuePair[0]

The figures for serializing 1,000 LogEvents at each operation are as follows. (See layout-benchmark directory for the full report.)

Benchmark	ops/sec⁶		B/op⁶
liteLogstashLayout4EcsLayout	813,346	▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (100%)	4,545.0
liteEcsLayout	603,469	▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (74%)	460.9
liteLogstashLayout4JsonLayout	564,456	▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (69%)	492.7
liteDefaultJsonLayout	294,590	▉▉▉▉▉▉▉ (36%)	5,101,398.2
liteCustomJsonLayout	258,302	▉▉▉▉▉▉▉ (32%)	5,516,833.5
fullLogstashLayout4JsonLayout	70,755	▉▉ (9%)	108,258.8
fullLogstashLayout4EcsLayout	41,913	▉ (5%)	35,690,881.3
fullEcsLayout	16,594	▉ (2%)	46,495,593.1
fullDefaultJsonLayout	8,484	▉ (1%)	234,422,375.7
fullCustomJsonLayout	8,456	▉ (1%)	234,624,867.7

⁶ 99^th percentile

Let us try to answer some common questions:

How come log4j2-logstash-layout can yield superior performance compared to Log4j 2 JsonLayout? Log4j 2 JsonLayout employs a single Jackson view to generate JSON, XML, and YAML outputs. For this purpose, it uses Jackson ObjectMapper, which needs to walk over the class fields via reflection and perform heavy branching and intermediate object instantiation. On the contrary, log4j2-logstash-layout parses the given template once and compiles an (almost) garbage- and (to a certain extent) branching-free JSON generator employing Jackson JsonGenerator.
Why is log4j2-logstash-layout is not totally garbage-free?
- Since Throwable#getStackTrace() clones the original StackTraceElement[], accesses to (and hence rendering) stack traces can never be garbage-free.
- Rendering of context data (that is, MDC) field values is garbage-free if the value is either null, or of type String, Short, Integer, Long, or byte[].
How can one run the benchmark on his/her machine? After a fresh mvn clean package within the source directory, run layout-benchmark/benchmark.py.

Contributors

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Name		Name	Last commit message	Last commit date
Latest commit History 181 Commits
layout-benchmark		layout-benchmark
layout-demo		layout-demo
layout-fatjar		layout-fatjar
layout		layout
.gitignore		.gitignore
.travis.yml		.travis.yml
CHANGELOG.md		CHANGELOG.md
COPYING.txt		COPYING.txt
README.md		README.md
pom.xml		pom.xml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

Table of Contents

Usage

Predefined Templates

Features

Fat JAR

Appender Support

Performance

Contributors

License

About

Uh oh!

Releases

Packages

Languages

License

wangzhuo0978/log4j2-logstash-layout

Folders and files

Latest commit

History

Repository files navigation

Table of Contents

Usage

Predefined Templates

Features

Fat JAR

Appender Support

Performance

Contributors

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages