Html escaping (#7445)

Add support to codecs and outputs to enable/disable escaping of html
symbols in JSON strings.

By default html escaping is enabled.

Author: Steffen Siering

CHANGELOG-developer.asciidoc

@@ -23,6 +23,7 @@ The list below covers the major changes between 6.3.0 and master only.
 - Moving of TLS helper functions and structs from `output/tls` to `tlscommon`. {pull}7054[7054]
 - Port fields.yml collector to Golang {pull}6911[6911]
 - Dashboards under _meta/kibana are expected to be decoded. See https://github.com/elastic/beats/pull/7224 for a conversion script. {pull}7265[7265]
+- Constructor `(github.com/elastic/beats/libbeat/output/codec/json).New` expects a new `escapeHTML` parameter. {pull}7445[7445]
 
 ==== Bugfixes
 

CHANGELOG.asciidoc

@@ -166,6 +166,7 @@ https://github.com/elastic/beats/compare/v6.2.3...master[Check the HEAD diff]
 - Add support for docker autodiscover to monitor containers on host network {pull}6708[6708]
 - Add ability to define input configuration as stringified JSON for autodiscover. {pull}7372[7372]
 - Add processor definition support for hints builder {pull}7386[7386]
+- Add support to disable html escaping in outputs. {pull}7445[7445]
 
 *Auditbeat*
 

auditbeat/auditbeat.reference.yml

@@ -335,6 +335,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 0
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional protocol and basic auth credentials.
   #protocol: "https"
   #username: "elastic"
@@ -440,6 +443,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 3
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional maximum time to live for a connection to Logstash, after which the
   # connection will be re-established.  A value of `0s` (the default) will
   # disable this feature.
@@ -554,6 +560,14 @@ output.elasticsearch:
   # Kafka version auditbeat is assumed to run against. Defaults to the "1.0.0".
   #version: '1.0.0'
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Metadata update configuration. Metadata do contain leader information
   # deciding which broker to use when publishing.
   #metadata:
@@ -656,6 +670,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # The list of Redis servers to connect to. If load balancing is enabled, the
   # events are distributed to the servers in the list. If one server becomes
   # unreachable, the events are distributed to the reachable servers only.
@@ -757,6 +779,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Path to the directory where to save the generated files. The option is
   # mandatory.
   #path: "/tmp/auditbeat"
@@ -784,8 +814,13 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
-  # Pretty print json event
-  #pretty: false
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
 
 #================================= Paths ======================================
 

filebeat/filebeat.reference.yml

@@ -977,6 +977,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 0
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional protocol and basic auth credentials.
   #protocol: "https"
   #username: "elastic"
@@ -1082,6 +1085,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 3
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional maximum time to live for a connection to Logstash, after which the
   # connection will be re-established.  A value of `0s` (the default) will
   # disable this feature.
@@ -1196,6 +1202,14 @@ output.elasticsearch:
   # Kafka version filebeat is assumed to run against. Defaults to the "1.0.0".
   #version: '1.0.0'
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Metadata update configuration. Metadata do contain leader information
   # deciding which broker to use when publishing.
   #metadata:
@@ -1298,6 +1312,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # The list of Redis servers to connect to. If load balancing is enabled, the
   # events are distributed to the servers in the list. If one server becomes
   # unreachable, the events are distributed to the reachable servers only.
@@ -1399,6 +1421,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Path to the directory where to save the generated files. The option is
   # mandatory.
   #path: "/tmp/filebeat"
@@ -1426,8 +1456,13 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
-  # Pretty print json event
-  #pretty: false
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
 
 #================================= Paths ======================================
 

heartbeat/heartbeat.reference.yml

@@ -444,6 +444,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 0
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional protocol and basic auth credentials.
   #protocol: "https"
   #username: "elastic"
@@ -549,6 +552,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 3
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional maximum time to live for a connection to Logstash, after which the
   # connection will be re-established.  A value of `0s` (the default) will
   # disable this feature.
@@ -663,6 +669,14 @@ output.elasticsearch:
   # Kafka version heartbeat is assumed to run against. Defaults to the "1.0.0".
   #version: '1.0.0'
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Metadata update configuration. Metadata do contain leader information
   # deciding which broker to use when publishing.
   #metadata:
@@ -765,6 +779,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # The list of Redis servers to connect to. If load balancing is enabled, the
   # events are distributed to the servers in the list. If one server becomes
   # unreachable, the events are distributed to the reachable servers only.
@@ -866,6 +888,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Path to the directory where to save the generated files. The option is
   # mandatory.
   #path: "/tmp/heartbeat"
@@ -893,8 +923,13 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
-  # Pretty print json event
-  #pretty: false
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
 
 #================================= Paths ======================================
 

libbeat/_meta/config.reference.yml

@@ -230,6 +230,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 0
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional protocol and basic auth credentials.
   #protocol: "https"
   #username: "elastic"
@@ -335,6 +338,9 @@ output.elasticsearch:
   # Set gzip compression level.
   #compression_level: 3
 
+  # Configure escaping html symbols in strings.
+  #escape_html: true
+
   # Optional maximum time to live for a connection to Logstash, after which the
   # connection will be re-established.  A value of `0s` (the default) will
   # disable this feature.
@@ -449,6 +455,14 @@ output.elasticsearch:
   # Kafka version beatname is assumed to run against. Defaults to the "1.0.0".
   #version: '1.0.0'
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Metadata update configuration. Metadata do contain leader information
   # deciding which broker to use when publishing.
   #metadata:
@@ -551,6 +565,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # The list of Redis servers to connect to. If load balancing is enabled, the
   # events are distributed to the servers in the list. If one server becomes
   # unreachable, the events are distributed to the reachable servers only.
@@ -652,6 +674,14 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
+
   # Path to the directory where to save the generated files. The option is
   # mandatory.
   #path: "/tmp/beatname"
@@ -679,8 +709,13 @@ output.elasticsearch:
   # Boolean flag to enable or disable the output module.
   #enabled: true
 
-  # Pretty print json event
-  #pretty: false
+  # Configure JSON encoding
+  #codec.json:
+    # Pretty print json event
+    #pretty: false
+
+    # Configure escaping html symbols in strings.
+    #escape_html: true
 
 #================================= Paths ======================================
 

libbeat/docs/outputconfig.asciidoc

@@ -138,6 +138,13 @@ Increasing the compression level will reduce the network usage but will increase
 
 The default value is 0.
 
+===== `escape_html`
+
+Configure escaping of HTML in strings. Set to `false` to disable escaping.
+
+The default value is `true`.
+
+
 ===== `worker`
 
 The number of workers per configured host publishing events to Elasticsearch. This
@@ -466,6 +473,12 @@ Increasing the compression level will reduce the network usage but will increase
 
 The default value is 3.
 
+===== `escape_html`
+
+Configure escaping of HTML in strings. Set to `false` to disable escaping.
+
+The default value is `true`.
+
 ===== `worker`
 
 The number of workers per configured host publishing events to Logstash. This
@@ -1187,13 +1200,16 @@ codec. By default the `json` codec is used.
 
 *`json.pretty`*: If `pretty` is set to true, events will be nicely formatted. The default is false.
 
+*`json.escape_html`*: If `escape_html` is set to false, html symbols will not be escaped in strings. The default is true.
+
 Example configuration that uses the `json` codec with pretty printing enabled to write events to the console:
 
 [source,yaml]
 ------------------------------------------------------------------------------
 output.console:
   codec.json:
     pretty: true
+    escape_html: false
 ------------------------------------------------------------------------------
 
 *`format.string`*: Configurable format string used to create a custom formatted message.