Fluent-bit crashes with SIGSEGV when adding Loki output #4072
Description
Bug Report
Describe the bug
Fluent-bit is configured to send logs to fluent-bit aggregator -> vector aggregator -> ES sink. This works fine until I add Loki sink to vector aggregator. It starts crashing. I tried outputting to Loki directly from fluent-bit, but SIGSEGV crashes happen.
- Example log message if applicable:
log="09/08/2021 10:13:29 message 12285424 - log message
" stream="stdout" time="2021-09-08T10:13:29.797700429Z
- Crashes with the following message:
2021/09/08 10:13:44] [engine] caught signal (SIGSEGV) 0x558a2762e9a2 in atomic_load_p() at lib/jemalloc-5.2.1/include/jemalloc/internal/atomic.h:62 0x558a2762e9a2 in extent_arena_get() at lib/jemalloc-5.2.1/include/jemalloc/internal/extent_inlines.h:51 0x558a2762e9a2 in je_large_dalloc() at lib/jemalloc-5.2.1/src/large.c:361 0x558a275f07ac in arena_dalloc_large() at lib/jemalloc-5.2.1/include/jemalloc/internal/arena_inlines_b.h:281 0x558a275f07ac in arena_dalloc() at lib/jemalloc-5.2.1/include/jemalloc/internal/arena_inlines_b.h:323 0x558a275f07ac in idalloctm() at lib/jemalloc-5.2.1/include/jemalloc/internal/jemalloc_internal_inlines_c.h:118 0x558a275f07ac in ifree() at lib/jemalloc-5.2.1/src/jemalloc.c:2589 0x558a275f07ac in je_free_default() at lib/jemalloc-5.2.1/src/jemalloc.c:2799 0x558a276b6a31 in flb_free() at include/fluent-bit/flb_mem.h:122 0x558a276b70d0 in modify_json_cond() at plugins/in_tail/tail_dockermode.c:171 0x558a276b76e5 in flb_tail_dmode_flush() at plugins/in_tail/tail_dockermode.c:353 0x558a276b78ed in file_pending_flush() at plugins/in_tail/tail_dockermode.c:408 0x558a276b7ace in flb_tail_dmode_pending_flush() at plugins/in_tail/tail_dockermode.c:460 0x558a2766c084 in flb_input_collector_fd() at src/flb_input.c:1004 0x558a2767bdca in flb_engine_handle_event() at src/flb_engine.c:352 0x558a2767bdca in flb_engine_start() at src/flb_engine.c:613 0x558a27661cd2 in flb_lib_worker() at src/flb_lib.c:493 0x7fbf26504fa2 in ???() at ???:0 0x7fbf25be54ce in ???() at ???:0 0xffffffffffffffff in ???() at ???:0
Expected behavior
Fluent-bit sends logs to Loki without crashes
Your Environment
- Version used: Fluent-bit 1.7.4 installed with HELM chart
- Configuration:
`podAnnotations:
fluentbit.io/exclude: "true"
vector.dev/exclude: "true"
image:
repository: fluent/fluent-bit
pullPolicy: Always
tag: 1.7.4
config:
service: |
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.service.port }}
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app.log
Exclude_Path .log.1,.gz,*.zip
DB /var/log/flb_kube.db
Parser docker
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser dockerMode_firstline
Mem_Buf_Limit 25MB
Skip_Long_Lines On
Refresh_Interval 10
Ignore_Older 25m
db.journal_mode MEMORY
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc.cluster.local:443
Merge_Log On
Merge_Log_Key data
K8S-Logging.Parser On
K8S-Logging.Exclude On
outputs: |
[OUTPUT]
name loki
match *
host instance-ip
port 3100
labels namespace=$kubernetes['namespace_name']
line_format key_value
auto_kubernetes_labels on
[PARSER]
Name dockerMode_firstline
Format regex
Regex /^.*\[(\d{4}-\d{1,2}-\d{1,2}.\d{2}:\d{2}:\d{2})(?<message>.*)\{/
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L`
- Environment name and version (e.g. Kubernetes? What version?): Amazon EKS v1.19.13
- Filters and plugins: Kubernetes filter
Additional context
Same configuration is used to forward logs to fluent-bit aggregator, then from there to vector-aggregator. No matter where I add Loki as output crashes happen.