FluentBit is hanging and stops collecting logs after a hot reload

[quanghungb]

Bug Report

Describe the bug
FluentBit is hanging and stops collecting logs after a hot reload that has been triggered.

Context
We observed 24 pods (of 46 pods of a DS) are presenting the same pattern logs

[2025/06/25 20:13:32] [engine] caught signal (SIGHUP)
[2025/06/25 20:13:32] [ info] reloading instance pid=1 tid=0x7f37941dae40
[2025/06/25 20:13:32] [ info] [reload] stop everything of the old context
[2025/06/25 20:13:32] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2025/06/25 20:13:32] [ info] [reload] start everything

The behaviour is very similar to what is reported here:

• Fluent-bit hanging and stopping operation #9927
• Hot reload stuck in progress after pausing inputs #9354 (but we don't have the info pause logs and it stucks after the "[ info] [reload] start everything"

FluentBit is hang, consumes almost no resources (CPU, memory) and no logs are collected.

It seems spending time on sleep

cat /proc/1548315/stack
[<0>] hrtimer_nanosleep+0x95/0x120
[<0>] common_nsleep+0x40/0x50
[<0>] __x64_sys_clock_nanosleep+0xc7/0x130
[<0>] do_syscall_64+0x35/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6

strace: Process 1548315 attached
restart_syscall(<... resuming interrupted clock_nanosleep ...>) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0
clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=1, tv_nsec=0}, 0x7ffe5b0605c0) = 0

Expected behavior
The reload should be done and logs to be collected continously

Your Environment

• Version used: 3.2.4
• Deployed by FluentBit Helm Chart fluent-bit-0.47.10 on Kubernetes v1.2.6

Additional context
We trigger the hot reload when the secret containing the certificate used by FluentBit kafka input (mtls required) has been updated.
It impacts 24 pods of 46. So huge impact on our logs collection pipelines :(

[pawel-lmcb]

@quanghungb question, can you post your config ? Also do you have threading turned on for inputs or workers for outputs ?

[quanghungb]

fluent-bit.yaml: |
    service:
      flush: 5
      grace: 5
      daemon: "off"
      dns.mode: UDP
      log_level: warn
      http_server: "on"
      http_listen: 0.0.0.0
      http_port: 2020
      coro_stack_size: 24576
      scheduler.cap: 2000
      scheduler.base: 5
      json.convert_nan_to_null: false
      sp.convert_from_str_to_num: true
      Health_Check: "on"
      Hot_Reload: "on"
    parsers:
      - format: regex
        name: kube-log-tag-parser
        regex: (?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>[^_]+)_(?<container_id>[a-z0-9]+)$
        skip_empty_values: true
        time_format: '%Y-%m-%dT%H:%M:%S.%L'
        time_keep: true
        time_key: time
        time_strict: true
    pipeline:
      inputs:
        - Tag: k8s.dmesg.*
          name: kmsg
          processors:
            logs:
              - action: insert
                key: kubernetes_cluster
                name: content_modifier
                value: ${KUBENERTES_CLUSTER}
              - action: insert
                key: kubernetes_host
                name: content_modifier
                value: ${KUBERNETES_NODE_NAME}
          alias: in.k8s-cluster-dmesg.kmsg.dmesg
        - Tag: kube.var.logs.<pod_name>_<namespace_name>_<container_name>_<container_id>
          Tag_Regex: .+/(?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]+)\.log$
          buffer_chunk_size: 128KB
          buffer_max_size: 128KB
          multiline.parser: cri, docker
          name: tail
          path: /var/log/containers/*.log
          processors:
            logs:
              - action: insert
                key: kubernetes_cluster
                name: content_modifier
                value: ${KUBENERTES_CLUSTER}
              - action: insert
                key: kubernetes_host
                name: content_modifier
                value: ${KUBERNETES_NODE_NAME}
              - Buffer_Size: 2M
                Cache_Use_Docker_Id: "On"
                Kube_Meta_Cache_TTL: "30"
                Kube_Tag_Prefix: kube.var.logs.
                Merge_Log: "On"
                Namespace_Annotations: "On"
                Namespace_Labels: "On"
                Regex_Parser: kube-log-tag-parser
                name: kubernetes
              - call: filter_ns
                name: lua
                script: /fluent-bit/scripts/filter_ns.lua
              - call: flatten_record
                name: lua
                script: /fluent-bit/scripts/flatten_record.lua
              - call: set_output_topic_pod_logs
                name: lua
                script: /fluent-bit/scripts/enrich_record.lua
          alias: in.k8s-pod-logs.tail.k8s-container-logs
      filters: []
      outputs:
        - name: kafka
          queue_full_retries: 0
          storage.total_limit_size: 100M
          rdkafka.request.required.acks: 1
          rdkafka.log.connection.close: false
          rdkafka.message.timeout.ms: 3600000
          rdkafka.batch.size: 10000000
          rdkafka.queue.buffering.max.ms: 10
          rdkafka.queue.buffering.max.kbytes: 10485760
          Brokers: broker:9094
          rdkafka.ssl.certificate.location: /ssl/prod/rp-user-fluentbit-prod/tls.crt
          Match: k8s.dmesg.*
          rdkafka.ssl.key.location: /ssl/prod/rp-user-fluentbit-prod/tls.key
          rdkafka.compression.codec: snappy
          alias: out.k8s-cluster-dmesg
          rdkafka.queue.buffering.max.messages: 10000
          Topics: k8s-cluster-dmesg
          rdkafka.ssl.ca.location: /ssl/ca/root-cacrt.crt
          rdkafka.security.protocol: ssl
        - name: kafka
          queue_full_retries: 0
          storage.total_limit_size: 100M
          rdkafka.request.required.acks: 1
          rdkafka.log.connection.close: false
          rdkafka.message.timeout.ms: 3600000
          rdkafka.batch.size: 10000000
          rdkafka.queue.buffering.max.ms: 10
          rdkafka.queue.buffering.max.kbytes: 10485760
          Brokers: broker:9094
          rdkafka.ssl.certificate.location: /ssl/sprod/rp-user-fluentbit-prod/tls.crt
          Match: kube.var.logs.*
          rdkafka.ssl.key.location: /ssl/prod/rp-user-fluentbit-prod/tls.key
          dynamic_topic: "On"
          rdkafka.compression.codec: snappy
          alias: out.k8s-pod-logs
          rdkafka.queue.buffering.max.messages: 10000
          Topics: k8s-other-pod-logs
          rdkafka.ssl.ca.location:/ssl/ca/root-cacrt.crt
          rdkafka.security.protocol: ssl
          topic_key: _tpaas_output_topic_

[patrick-stephens]

Is this PR related? #10547

[quanghungb]

@patrick-stephens your fix is only for v4 series?

[patrick-stephens]

It looks pretty simple to backport to the 3.2 branch if you want to make a PR?

I'm not sure if or when it'll make a release though for the older versions.

[CharlieR-o-o-t]

Hello, this seems duplicate of: #10670

I have fixed it with PR: #10671
Awaiting merge.

@patrick-stephens , #10547 will not fix this behavior.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.