Logs duplicated when fluentbit read from mountvolume

[daVidy31]

Bug Report

Describe the bug

Pods of fluentbit duplicate the same log from the same path with the tail plugin

To Reproduce

• kubernetes

• fluentbit helm chart (daemonset)

• elastic search

• Steps to reproduce the problem:
  In my kubernetes cluster are 3 nodes. So i have 3 fluentbit pods that are running

In the values.yaml of the fluentbit chart, i configured a volume and volumemount

In that path are logs.
When I read with fluentbit the logs, with tail plugin, and output them with es.

I have the replication of logs (as number of pods of fluentbit)

if is 2 pods - duplicated. I tried with only 1 fluentbit pod, and they did not duplicated!

Expected behavior
The logs will not be duplicated in the elasticsearch/,kibana

Code

extraVolumes:

• name: volume
  persistentVolumeClaim:
  claimName: x-pvc

extraVolumeMounts:

• name: volume
  subPath: /logs/x/
  mountPath: /logs/x/
• name: volume
  subPath: l/ogs/y/
  mountPath: /logs/y/

config:
service: |

[SERVICE]
    Daemon Off
    Flush x
    Log_Level x
    Parsers_File /fluent-bit/etc/parsers.conf
    Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
    HTTP_Server On
    HTTP_Listen x
    HTTP_Port x
    Health_Check On

inputs: |

[INPUT]
    Name tail
    Path /logs/y*
    Tag logs
    Mem_Buf_Limit 100MB
    Skip_Long_Lines Off
    Refresh_Interval 30
    Path_Key filename
    Skip_Empty_Lines On

[INPUT]
    Name tail
    Path /logs/x/*
    Tag logs
    Mem_Buf_Limit 100MB
    Skip_Long_Lines Off
    Refresh_Interval 30
    Path_Key filename
    Skip_Empty_Lines On

filters: |

[FILTER]
    name multiline
    match *
    multiline.key_content log
    multiline.parser java

[FILTER]
    Name modify
    Match *
    Add cluster_name $var
    Add cluster_id $var

outputs: |

[OUTPUT]
    Name es
    Match logs
    Host ip
    http_user     x
    http_passwd   y
    tls On
    tls.verify off
    Logstash_Format off
    Index logs
    Logstash_DateFormat %Y.%m
    Suppress_Type_Name On

customParsers: |

[PARSER]
    Name docker_no_time
    Format json
    Time_Keep Off
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L

[PARSER]
    Name log_parser
    Format regex
    Regex \[(?<timestamp>.*)\]\[(?<correlationID>.*)\]\[(?<severity>.*)\]\[(?<class>.*)\]\[(?<thread>.*)\] (?<text>.*)

[PARSER]
    Name equipment_parser
    Format regex
    Regex ^(?<log>.*)$

[PARSER]
    Name kafka_parser
    Format regex
    Regex \[(?<timestamp>.*)\] (?<severity>FATAL|ERROR|WARN|INFO|DEBUG|TRACE) (?<text>.*)

[PARSER]
    Name postgres_parser
    Format regex
    Regex ^(?<timestamp>.{23}).*(?<severity>FATAL|ERROR|WARN|INFO|DEBUG|TRACE|LOG|DETAIL): (?<text>.*)

Additional context
I try to show the logs from a specifc path from a volume!

[patrick-stephens]

If you're mounting the same files into each pod and telling both pods to read those files then yes it will read both files once per pod - that's what you've told it to do. You need to restrict the files you want read by each pod.

[daVidy31]

Hello, how to restrict, in the values.yaml?

[patrick-stephens]

Set one to read logs/x and one to read logs/y - there's probably a few ways to do that (e.g. some kind of ID for the sub-directory passed in as an env var for the config, maybe change the mounts on each pod, etc.). It's hard to say without knowing your specific requirements and probably specific to them.

This is an X-Y problem I think, nothing to do with Fluent Bit but how to configure replica pods with different values.

[daVidy31]

Hi, the problem is not in the path, I need both path to be read by the fluentbit(only by one replica). Somehow to restrict other pods to read the same logs.

[patrick-stephens]

Configure it so only one pod does it then. It's the same issue as using a replica where you want only one pod to do something.
You could do something like setting the match via an environment variable or something. You could do leader election. You could do something else.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.