Fluent Bit Multiline Java Parsing is Not Working as Expected for WebSphere Java Logs #9407
Comments
|
Please provide the full details in the template including things like the target, whether you're using K8S, etc. Please provide your full configuration, i.e. are you using the recommended multiline parsers for docker or cri prior to attempting multiline parsing? e.g. https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-and-containers-v1.8 if you are on K8S which it looks like you are. Are the failing lines partial ones from the kubelet? Identified by Please update to the latest version as well to pick up any fixes there and retest. |
|
Hi @patrick-stephens thanks for checking this. I have updated the description with • Version used: 2.2.3 We are using java multiline build in parser. Please let us know if you need other information to investigate. |
Bug Report
Describe the bug
The Fluent Bit multiline Java parser is not stitching together logs as expected for WebSphere Java logs. It appears that some line in the WebSphere logs is treated as a new log entry, despite the configuration for multiline logging.
To Reproduce
Set up Fluent Bit with the default Java multiline parser.
Generate logs from a WebSphere Java application where each log spans multiple lines.
Observe that some lines are treated as a separate log entry rather than being stitched together.
Expected behavior
Logs are stitched together without breaking wording
Screenshots
Your Environment
[SERVICE]
#Default service flush interval is 15 seconds
Flush 15
HTTP_Server Off
Daemon Off
storage.path /var/opt/microsoft/docker-cimprov/state/flbstore/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 10M
Log_Level info
Parsers_File /etc/opt/microsoft/docker-cimprov/azm-containers-parser.conf
Log_File /var/opt/microsoft/docker-cimprov/log/fluent-bit.log
plugins_file /etc/opt/microsoft/docker-cimprov/azm-containers-input-plugins.conf
@include fluent-bit-input.conf
[INPUT]
Name tail
Alias oms_tail
Tag oms.container.log.la.*
Path ${AZMON_LOG_TAIL_PATH}
Read_from_Head true
DB /var/log/omsagent-fblogs.db
DB.Sync Off
Parser cri
Mem_Buf_Limit 10m
Buffer_Chunk_Size 1m
Buffer_Max_Size 1m
Rotate_Wait 20
Refresh_Interval 30
Path_Key filepath
Skip_Long_Lines On
Exclude_Path ${AZMON_CLUSTER_LOG_TAIL_EXCLUDE_PATH}
[FILTER]
Name multiline
Match oms.container.log.la.*
multiline.key_content log
multiline.parser "java"
[FILTER]
Name grep
Alias oms_filter
Match oms.container.log.la.*
Exclude stream ${AZMON_LOG_EXCLUSION_REGEX_PATTERN}
@include fluent-bit-common.conf
@include fluent-bit-internal-metrics.conf
Additional context
The issue was discovered during log collection from WebSphere. Despite setting up multiline parsing in Fluent Bit, logs are still being split into separate records for some lines. The issue may be related to how WebSphere generates the logs (starting a new entry for each line) or a bug in the default Java multiline parser.
We have identified the following log entries that are split across multiple records in the ContainerLogV2 table:
ContainerLogV2
| where ContainerName contains 'sorl'
and TimeGenerated between (datetime('2024-09-10T15:20:57') .. datetime('2024-09-10T15:21:00') )
and LogMessage !contains 'readiness'
| order by TimeGenerated desc
The text was updated successfully, but these errors were encountered: