Filebeat Elasticsearch module ingest pipelines fail to parse deprecation logs #26857
Closed
Description
Overview
For confirmed bugs, please report:
- Version: 7.13.3
- Operating System: MacOSX
Discuss Forum URLSDH- Steps to Reproduce: see below
The Filebeat Elasticsearch module ingest pipelines fails to parse deprecations logs, both in json and plaintext format. The consequence is that these logs are not searchable Kibana using the standard index pattern due to:
- Json logs being dropped
- Plaintext logs missing
@timestamp
My guess as to the root cause is that the Elasticsearch deprecation logs have changed in format and the module has not been updated to keep up.
Steps to reproduce
- Install Elasticsearch, Kibana, Filebeat 7.13.3
- Apply the below filebeat.yml and elasticsearch.yml module config ( with your log path)
- Start Filebeat
filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.elasticsearch:
hosts: ["localhost:9200"]
#output.file:
# path: "/tmp/"
# filename: filebeat-es-deprecation
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
elasticsearch.yml
- module: elasticsearch
server:
enabled: false
gc:
enabled: false
audit:
enabled: false
slowlog:
enabled: false
deprecation:
enabled: true
var.paths:
- /Users/mattias/dev/elastic_stack/elasticsearch-7.13.3/logs/*deprecation.*
Initial search:
GET filebeat-7.13.3/_search
{
"size": 0,
"aggs": {
"logs.file.path": {
"terms": {
"field": "log.file.path",
"size": 10
},
"aggs": {
"has_error": {
"filter": {
"exists": {
"field": "error.message"
}
}
}
}
}
}
}
Response ( truncated )
"aggregations" : {
"logs.file.path" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "/Users/mattias/dev/elastic_stack/elasticsearch-7.13.3/logs/elasticsearch_deprecation.log",
"doc_count" : 3,
"has_error" : {
"doc_count" : 3
}
}
]
}
}
The response shows two issues:
- All logs entries from the plaintext log file have errors
- No logs can be found from the json log file
Plaintext log issue
After a second search, all my log entries have grok errors.
Query:
GET filebeat-7.13.3/_search?filter_path=hits.hits._source
{
"_source": ["error.message"]
}
Response (truncated):
{
"hits": {
"hits": [
{
"_source": {
"error": {
"message": "Provided Grok expressions do not match field value: [[2021-07-09T10:33:11,962][DEPRECATION][o.e.d.r.RestController ] [Mattiass-MacBook-Pro.local] Legacy index templates are deprecated in favor of composable templates.]"
}
}
},
...
]
}
}
JSON log issue
Log messages not coming in, I found the issue in the ingest pipeline. Steps to reproduce:
1. Enable file output
Commented ES output and inserted
output.file:
path: "/tmp/"
filename: filebeat-es-deprecation
Cleared data directory and restarted FB
2 Simulate request
Using one of the events from the file output, I created simulate request ( I have redacted my mac address and IPs ):
POST _ingest/pipeline/filebeat-7.13.3-elasticsearch-deprecation-pipeline/_simulate?verbose
{
"docs": [
{
"_source": {
"@timestamp": "2021-07-09T01:20:10.075Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.13.3",
"pipeline": "filebeat-7.13.3-elasticsearch-deprecation-pipeline"
},
"agent": {
"ephemeral_id": "bdf967c4-c98b-42c5-9e08-fb052dd213a0",
"id": "ed3e1bb2-bbb7-4499-b2e4-6ea268b64209",
"name": "Mattiass-MacBook-Pro.local",
"type": "filebeat",
"version": "7.13.3",
"hostname": "Mattiass-MacBook-Pro.local"
},
"ecs": {
"version": "1.9.0"
},
"host": {
"architecture": "x86_64",
"os": {
"version": "10.16",
"family": "darwin",
"name": "Mac OS X",
"kernel": "20.3.0",
"build": "20D91",
"type": "macos",
"platform": "darwin"
},
"id": "B18111C0-F268-5C25-8165-DADD4BF58545",
"hostname": "Mattiass-MacBook-Pro.local",
"name": "Mattiass-MacBook-Pro.local"
},
"message": """{"type": "deprecation.elasticsearch", "timestamp": "2021-07-09T10:42:03,065+10:00", "level": "DEPRECATION", "component": "o.e.d.c.m.MetadataCreateIndexService", "cluster.name": "elasticsearch", "node.name": "Mattiass-MacBook-Pro.local", "message": "index name [.kibana-event-log-7.13.3-000001] starts with a dot '.', in the next major version, index names starting with a dot are reserved for hidden indices and system indices", "cluster.uuid": "k_63KpgPSDeFRu1NxnlDiw", "node.id": "VLed1qvSSOCitAo-UeM3Jg" }""",
"log": {
"offset": 390,
"file": {
"path": "/Users/mattias/dev/elastic_stack/elasticsearch-7.13.3/logs/elasticsearch_deprecation.json"
}
},
"event": {
"module": "elasticsearch",
"dataset": "elasticsearch.deprecation"
},
"fileset": {
"name": "deprecation"
},
"service": {
"type": "elasticsearch"
},
"input": {
"type": "log"
}
}
}
]
}
The response shows that the event was dropped as per the below excerpt, due to the the elasticsearch.deprecation.type not matching. ( Real value deprecation.elasticsearch, expected value deprecation )
...
{
"processor_type" : "json",
"status" : "success",
"doc" : {
"_index" : "_index",
"_type" : "_doc",
"_id" : "_id",
"_source" : {
...
"elasticsearch" : {
"deprecation" : {
"node.id" : "VLed1qvSSOCitAo-UeM3Jg",
"cluster.name" : "elasticsearch",
"component" : "o.e.d.c.m.MetadataCreateIndexService",
"level" : "DEPRECATION",
"node.name" : "Mattiass-MacBook-Pro.local",
"type" : "deprecation.elasticsearch",
...
{
"processor_type" : "drop",
"status" : "dropped",
"if" : {
"condition" : "ctx.elasticsearch.deprecation.type != 'deprecation'",
"result" : true
}
}