Fluent bit hangs, any step to troubleshoot?

[yhzs8]

Bug Report

Describe the bug
Fluent bit hangs, any step to troubleshoot?

To Reproduce
Install and run Fluent Bit v3.1.7 on RHEL8.10
It is expected to run continuously

Expected behavior
It hangs randomly after 20min/2h

Your Environment

• Version used: 3.1.7
• Configuration:

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        5

    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level   info

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    # plugins_file plugins.conf

    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020

    # Storage
    # =======
    # Fluent Bit can use memory and filesystem buffering based mechanisms
    #
    # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
    #
    # storage metrics
    # ---------------
    # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
    # exported only if the 'http_server' option is enabled.
    #
    storage.metrics on

    # storage.path
    # ------------
    # absolute file system path to store filesystem data buffers (chunks).
    #
    # storage.path /tmp/storage

    # storage.sync
    # ------------
    # configure the synchronization mode used to store the data into the
    # filesystem. It can take the values normal or full.
    #
    # storage.sync normal

    # storage.checksum
    # ----------------
    # enable the data integrity check when writing and reading data from the
    # filesystem. The storage layer uses the CRC32 algorithm.
    #
    # storage.checksum off

    # storage.backlog.mem_limit
    # -------------------------
    # if storage.path is set, Fluent Bit will look for data chunks that were
    # not delivered and are still in the storage layer, these are called
    # backlog data. This option configure a hint of maximum value of memory
    # to use when processing these records.
    #
    # storage.backlog.mem_limit 5M

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag                <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              /var/log/secure
    Tag               input.os.secure
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    path              /var/log/audit/audit.log
    Tag               input.os.audit
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              /var/log/boot.log
    Tag               input.os.boot
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path             <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15
    multiline.parser  python-multiline-regex

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15
    multiline.parser  python-multiline-regex

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15
    multiline.parser  python-multiline-regex

[INPUT]
    Name              tail
    Path_Key          filename
    Path              <obfuscated>
    Tag               <obfuscated>
    Inotify_Watcher   False
    DB                /var/log/fluentbit_db_status.db
    Read_from_Head    On
    Skip_Long_Lines   On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 64KB
    Buffer_Max_Size   256KB
    Mem_Buf_Limit     10MB
    Refresh_Interval  15

################################
# filters for <obfuscated>
################################
[FILTER]
    Name         parser
    Match        <obfuscated>
    Key_Name     log
    Preserve_Key On
    Reserve_Data On
    Parser       syslog-rfc3164

[FILTER]
    Name   lua
    Match  <obfuscated>
    script /etc/fluent-bit/scripts/append.lua
    call   append_tag

[FILTER]
    Name   lua
    Match  <obfuscated>
    script /etc/fluent-bit/scripts/append.lua
    call   append_unix_time_for_rfc3164_as_utc

[FILTER]
    Name   record_modifier
    Match  <obfuscated>
    Record category operations

############################
# filters for input.os.audit
############################
[FILTER]
    Name         parser
    Match        input.os.audit
    Key_Name     log
    Preserve_Key On
    Reserve_Data On
    Parser       auditlog

[FILTER]
    Name   lua
    Match  input.os.audit
    script /etc/fluent-bit/scripts/append.lua
    call   append_tag

[FILTER]
    Name   lua
    Match  input.os.audit
    script /etc/fluent-bit/scripts/append.lua
    call   append_time_as_number

[FILTER]
    Name   record_modifier
    Match  input.os.audit
    Record category security

###########################
# filters for input.os.boot
###########################
[FILTER]
    Name   lua
    Match  input.os.boot
    script /etc/fluent-bit/scripts/append.lua
    call   append_tag_and_time

[FILTER]
    Name   record_modifier
    Match  input.os.boot
    Record category operations

#############################
# filters for input.os.secure
#############################
[FILTER]
    Name         parser
    Match        input.os.secure
    Key_Name     log
    Preserve_Key On
    Reserve_Data On
    Parser       syslog-rfc3164

[FILTER]
    Name   lua
    Match  input.os.secure
    script /etc/fluent-bit/scripts/append.lua
    call   append_tag

[FILTER]
    Name   lua
    Match  input.os.secure
    script /etc/fluent-bit/scripts/append.lua
    call   append_unix_time_for_rfc3164

[FILTER]
    Name   record_modifier
    Match  input.os.secure
    Record category security

######################################
# filters for <obfuscated>
######################################
[FILTER]
    Name         parser
    Match        <obfuscated>
    Key_Name     log
    Preserve_Key On
    Reserve_Data On
    Parser      <obfuscated>

[FILTER]
    Name   lua
    Match  <obfuscated>
    script /etc/fluent-bit/scripts/append.lua
    call   append_tag

[FILTER]
    Name   lua
    Match  <obfuscated>
    script /etc/fluent-bit/scripts/append.lua
    call   <obfuscated>

[FILTER]
    Name   record_modifier
    Match  <obfuscated>
    Record category operations

<several filters for our applications>

[OUTPUT]
    Name          http
    Match         input.*
    Host          <IP>
    Port          8081
    Header        <obfuscated>
    URI           <obfuscated>
    format        json
    json_date_key false
    Workers       2
    Retry_Limit   no_limits
    compress      gzip

[OUTPUT]
    name  stdout
    match **

• Environment name and version: Baremetal installation on RHEL8.10

• Server type and version: N/A

• Operating System and version: RHEL8.10

• Filters and plugins: See above

Additional context
The fluent-bit was totally hanging, service fluent-bit restart times out:

Nov 20 11:43:27 <obfuscated>fluent-bit[49564]: [2024/11/20 11:43:27] [engine] caught signal (SIGTERM)
Nov 20 11:44:57 <obfuscated>systemd[1]: fluent-bit.service: State 'stop-sigterm' timed out. Killing.
Nov 20 11:44:57 <obfuscated>systemd[1]: fluent-bit.service: Killing process 49564 (fluent-bit) with signal SIGKILL.
Nov 20 11:46:28 <obfuscated>systemd[1]: fluent-bit.service: Processes still around after SIGKILL. Ignoring.
Nov 20 11:47:58 <obfuscated>systemd[1]: fluent-bit.service: State 'stop-final-sigterm' timed out. Killing.
Nov 20 11:47:58 <obfuscated>systemd[1]: fluent-bit.service: Killing process 49564 (fluent-bit) with signal SIGKILL.
Nov 20 11:49:28 <obfuscated>systemd[1]: fluent-bit.service: Processes still around after final SIGKILL. Entering failed mode.
Nov 20 11:49:28 <obfuscated>systemd[1]: fluent-bit.service: Failed with result 'timeout'.
Nov 20 11:49:28 <obfuscated>systemd[1]: Stopped Fluent Bit.

fluent-bit dump at the time it was hanging:

Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: [2024/11/20 15:13:27] [engine] caught signal (SIGCONT)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: [2024/11/20 15:13:27] Fluent Bit Dump
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ===== Input =====
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.0 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.1 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.2 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.3 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.4 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.5 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 9.4K (9663 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 1
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 1
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.6 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.7 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 479b (479 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 1
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 1
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.8 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.9 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.10 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.11 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.12 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.13 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.14 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.15 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: tail.16 (tail)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ status
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ overlimit     : no
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     ├─ mem size   : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │     └─ mem limit  : 9.5M (10000000 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ tasks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ total tasks   : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ new           : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  ├─ running       : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │  └─ size          : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: │
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ chunks
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ total chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ up chunks  : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       ├─ down chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:       └─ busy chunks: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          ├─ size    : 0b (0 bytes)
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:          └─ size err: 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ===== Storage Layer =====
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: total chunks     : 2
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: ├─ mem chunks    : 2
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]: └─ fs chunks     : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    ├─ up         : 0
Nov 20 15:13:27 <obfuscated> fluent-bit[1551]:    └─ down       : 0

Do we have any advice how to proceed with troubleshooting?

[patrick-stephens]

Any specific customisation or lockdown to your node?
Can you try the latest version?
You can increase the log_level as well to debug.
Your tail config is a little strange, any reason you're disabling inotify.
Are the logs definitely updating? What about the DB file - is it full?
Ensure there is no overlap in the logs for each input as you're using the same DB for them all - potentially you can do it with less tail inputs using a tag_regex to label them too.
Can you see what connections are being made and/or traffic from the http output?
What's the net.core.somaxconn?
Does it work on RHEL 9 (later kernel)?
Can you remove outputs/simplify to a smaller reproducer or does the problem go away without a specific input or output?
What do your LUA scripts do? If you remove them does it still fail?

[yhzs8]

Any specific customisation or lockdown to your node?

Not really

Can you try the latest version?

Yes using 3.2.1 now

You can increase the log_level as well to debug.

Yes turned to debug now

Your tail config is a little strange, any reason you're disabling inotify.

Will check with app developer

Are the logs definitely updating? What about the DB file - is it full?

The DB file is just sqlite file, can you explain how it could be full?
I checked the in_tail_files table only contains 6 rows, each of the row represent the log files we are monitoring, and the timestamp conlumn point to the time when it hangs.

Ensure there is no overlap in the logs for each input as you're using the same DB for them all - potentially you can do it with less tail inputs using a tag_regex to label them too.

Can you elaborate what is overlap in the logs for each input,

Can you see what connections are being made and/or traffic from the http output?

All the connections before the hanging was properly responded with HTTP 202 response code

What's the net.core.somaxconn?

2048

Does it work on RHEL 9 (later kernel)?

Cannot try as our requirement is RHEL8.

Can you remove outputs/simplify to a smaller reproducer or does the problem go away without a specific input or output?
What do your LUA scripts do? If you remove them does it still fail?

This is our next step, first see if we can get any clue from debug log level and latest fluent-bit version.

[yhzs8]

So with fluent bit 3.2.1, and using debug, saw the following printed out before it hangs:

Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [0] input.application.<obfuscated>: [[1732542268.726537466, {}], {"time"=>1732560268, "log"=>"<log content>"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [1] input.application.<obfuscated>: [[1732542268.726540088, {}], {"time"=>1732560268, "log"=>"<log content>"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [0] input.application.<obfuscated>: [[1732542270.226547718, {}], {"log"=>"<log content>
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: ", "tag"=>"input.application.<obfuscated>", "time"=>1732542270, "filename"=>"/var/log/<obfuscated>.log", "category"=>"operations"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [1] input.application.<obfuscated>: [[1732542270.476666688, {}], {"log"=>"<log content>
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: ", "tag"=>"input.application.<obfuscated>", "time"=>1732542270, "filename"=>"/var/log/<obfuscated>.log", "category"=>"operations"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2] input.application.<obfuscated>: [[1732542270.726553440, {}], {"log"=>"<log content>
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: ", "tag"=>"input.application.<obfuscated>", "time"=>1732542270, "filename"=>"/var/log/<obfuscated>.log", "category"=>"operations"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [3] input.application.<obfuscated>: [[1732542270.726568222, {}], {"log"=>"<log content>
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: ", "tag"=>"input.application.<obfuscated>", "time"=>1732542270, "filename"=>"/var/log/<obfuscated>.log", "category"=>"operations"}]
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [task] created task=0x7f46efba5f20 id=0 OK
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [out flush] cb_destroy coro_id=7690
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [output:http:http.0] task_id=0 assigned to thread #0
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [upstream] KA connection #277 to <obfuscated>:8081 has been assigned (recycled)
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [upstream] KA connection #280 to <obfuscated>:8081 has been assigned (recycled)
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [output:stdout:stdout.1] task_id=0 assigned to thread #0
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [http_client] not using http_proxy for header
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [task] created task=0x7f46efba5fc0 id=1 OK
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [output:http:http.0] task_id=1 assigned to thread #1
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [output:stdout:stdout.1] task_id=1 assigned to thread #0
Nov 25 08:44:30 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:30] [debug] [out flush] cb_destroy coro_id=7691
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [ info] [output:http:http.0] <obfuscated>:8081, HTTP status=202
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: {<response_body>}
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [ info] [output:http:http.0] <obfuscated>:8081, HTTP status=202
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: {<response_body>}
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [debug] [upstream] KA connection #277 to <obfuscated>:8081 is now available
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [debug] [upstream] KA connection #280 to <obfuscated>:8081 is now available
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [debug] [out flush] cb_destroy coro_id=3987
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [debug] [task] destroy task=0x7f46efba5f20 (task_id=0)
Nov 25 08:44:31 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:31] [debug] [task] destroy task=0x7f46efba5fc0 (task_id=1)
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [task] created task=0x7f46efba5de0 id=0 OK
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [output:http:http.0] task_id=0 assigned to thread #0
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [output:stdout:stdout.1] task_id=0 assigned to thread #0
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [0] input.application.<obfuscated>: [[1732542272.726534366, {}], {"time"=>1732560272, "log"=>"<log_content>"}]
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [1] input.application.<obfuscated>: [[1732542272.726536750, {}], {"time"=>1732560272, "log"=>"<log_content>"}]
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [out flush] cb_destroy coro_id=7692
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [upstream] KA connection #279 to <obfuscated>:8081 has been assigned (recycled)
Nov 25 08:44:35 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:35] [debug] [http_client] not using http_proxy for header
Nov 25 08:44:36 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:36] [ info] [output:http:http.0] <obfuscated>:8081, HTTP status=202
Nov 25 08:44:36 <obfuscated> fluent-bit[1541]: {<response_body>}
Nov 25 08:44:36 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:36] [debug] [task] destroy task=0x7f46efba5de0 (task_id=0)
Nov 25 08:44:36 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:36] [debug] [upstream] KA connection #279 to <obfuscated>:8081 is now available
Nov 25 08:44:36 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:36] [debug] [out flush] cb_destroy coro_id=3988
Nov 25 08:44:46 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:46] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)
Nov 25 08:44:52 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:52] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)
Nov 25 08:44:52 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:52] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)
Nov 25 08:44:56 <obfuscated> fluent-bit[1541]: [2024/11/25 08:44:56] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)
Nov 25 08:45:01 <obfuscated> fluent-bit[1541]: [2024/11/25 08:45:01] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)
Nov 25 08:45:07 <obfuscated> fluent-bit[1541]: [2024/11/25 08:45:07] [debug] [upstream] drop keepalive connection #-1 to <obfuscated>:8081 (keepalive idle timeout)

Verified that the :8081 works to send HTTP request.

[patrick-stephens]

I appreciate the requirement is it must run on RHEL 8 but the main point of trying RHEL 9 is to see if it is a kernel or other parameter issue.

What are the ulimit and sysctl values for fs.inotify?
What are your connection counts, etc.? Are they near or at limits?

You could try upping those (e.g. some examples below but will not be maintained over reboot):

sudo sysctl fs.inotify.max_user_watches=65536
sudo sysctl fs.inotify.max_user_instances=65536
sudo tee /etc/systemd/system/fluent-bit.d/override.conf << EOF
[Service]
LimitNOFILE=65536
EOF
sudo systemctl daemon-reload
sudo systemctl restart fluent-bit

[yhzs8]

What are the ulimit and sysctl values for fs.inotify?

ulimit -n
1024
ulimit -u
865526
ulimit -i
865526

cat /proc/1540/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        unlimited            unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             865526               865526               processes
Max open files            1024                 262144               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       865526               865526               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

ls -l /proc/1540/fd | wc -l
278

sysctl fs.inotify
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 1048576

where PID 1540 is the fluent-bit process currently hanging.

[yhzs8]

@patrick-stephens works fine (non hanging after several days) after applying the change according to your suggestion.

[yhzs8]

@patrick-stephens a follow up question:
Since all our [INPUT] configuration is using file stat watchers instead of inotify watchers:

Inotify_Watcher   False

do we still need to change the sysctl ?

sudo sysctl fs.inotify.max_user_watches=65536
sudo sysctl fs.inotify.max_user_instances=65536

Will it work with only the:

[Service]
LimitNOFILE=65536

change?

[yhzs8]

@patrick-stephens any update?