[input:tail:tail.0] cannot read info from: /path/to/file/traffic*.log

[raviganapatp]

Bug Report

Describe the bug
i have a file and writing logs to it continuously. "Tail" input plugin is not able to read the file

To Reproduce
I have deployed fluent-bit using helm chart. Below is my values.yaml

dashboards:
  enabled: false
  labelKey: grafana_dashboard
  labelValue: 1
  annotations: {}
  namespace: ""
  deterministicUid: false

image:
  repository: cr.fluentbit.io/fluent/fluent-bit

logLevel: trace

## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  #inputs: |
  #  [INPUT]
  #      Name tail
  #      Path /var/log/containers/*.log
  #      multiline.parser docker, cri
  #      Tag kube.*
  #      Mem_Buf_Limit 5MB
  #      Skip_Long_Lines On
#
  #  [INPUT]
  #      Name systemd
  #      Tag host.*
  #      Systemd_Filter _SYSTEMD_UNIT=kubelet.service
  #      Read_From_Tail On

  #        Parser      traffic_parser
  inputs: |
    [INPUT]
        Name        tail
        Path        /var/log/longrun/traffic*.log
        Tag         traffic
        Mem_Buf_Limit 5MB
        DB          /var/log/longrun/trf.db
        Buffer_Max_Size 10M
        Buffer_Chunk_Size 2M

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name parser
        Match traffic
        Key_Name msg
        Parser traffic_parser

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  #outputs: |
  #  [OUTPUT]
  #      Name es
  #      Match kube.*
  #      Host elasticsearch-master-hl
  #      Logstash_Format On
  #      Retry_Limit False
  #      Suppress_Type_Name On
#
  #  [OUTPUT]
  #      Name es
  #      Match host.*
  #      Host elasticsearch-master-hl
  #      Logstash_Format On
  #      Logstash_Prefix node
  #      Retry_Limit False
  #      Suppress_Type_Name On
  outputs: |
    [OUTPUT]
        Name es
        Match traffic
        Host elasticsearch-master-hl
        Retry_Limit False
        Suppress_Type_Name On

    [OUTPUT]
        Name stdout
        Match *
        
  ## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/upstream-servers
  ## This configuration is deprecated, please use `extraFiles` instead.
  upstream: {}

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

    [PARSER]
        Name traffic_parser
        Format regex
        Regex ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} - Endpoint: https?:\/\/[^\s]+ - (Status: \d{3}|Error: .+)$
        Time_Keep Off
        Time_Key timestamp
        Time_Format %Y-%m-%dT%H:%M:%S,%L

  # This allows adding more files with arbitrary filenames to /fluent-bit/etc/conf by providing key/value pairs.
  # The key becomes the filename, the value becomes the file content.
  #extraFiles: 
  #  traffic.conf: | 
  #    [INPUT]
  #        Name tail
  #        Path /var/log/longrun/endpoint-responses.log
  #        Tag traffic.*
  #        Parser traffic_parser
  #        Mem_Buf_Limit 5MB
  #        Skip_Long_Lines On
#
  #    [OUTPUT]
  #        Name es
  #        Match traffic.*
  #        Host elasticsearch-master-hl
  #        Index traffic_metrics
  #        Retry_Limit False
  #        Suppress_Type_Name On
  #        Logstash_Format    On
  #        Logstash_Prefix    traffic_metrics

daemonSetVolumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: etcmachineid
    hostPath:
      path: /etc/machine-id
      type: File

daemonSetVolumeMounts:
  - name: varlog
    mountPath: /var/log
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true
  - name: etcmachineid
    mountPath: /etc/machine-id
    readOnly: true

command:
  - /fluent-bit/bin/fluent-bit

args:
  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.conf

• Fluent-bit pod log :

Fluent Bit v3.2.1
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _ _ ______ _ _ _____ _____
| ___| | | | | ___ (_) | |____ |/ __ \
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __ / /`' / /'
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / \ \ / /
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /.___/ /./ /___
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ \____(_)_____/


[2024/11/22 11:43:06] [ info] Configuration:
[2024/11/22 11:43:06] [ info] flush time | 1.000000 seconds
[2024/11/22 11:43:06] [ info] grace | 5 seconds
[2024/11/22 11:43:06] [ info] daemon | 0
[2024/11/22 11:43:06] [ info] ___________
[2024/11/22 11:43:06] [ info] inputs:
[2024/11/22 11:43:06] [ info] tail
[2024/11/22 11:43:06] [ info] ___________
[2024/11/22 11:43:06] [ info] filters:
[2024/11/22 11:43:06] [ info] kubernetes.0
[2024/11/22 11:43:06] [ info] parser.1
[2024/11/22 11:43:06] [ info] ___________
[2024/11/22 11:43:06] [ info] outputs:
[2024/11/22 11:43:06] [ info] es.0
[2024/11/22 11:43:06] [ info] stdout.1
[2024/11/22 11:43:06] [ info] ___________
[2024/11/22 11:43:06] [ info] collectors:
[2024/11/22 11:43:06] [ info] [fluent bit] version=3.2.1, commit=600b5a955b, pid=1
[2024/11/22 11:43:06] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/11/22 11:43:06] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/11/22 11:43:06] [ info] [simd ] disabled
[2024/11/22 11:43:06] [ info] [cmetrics] version=0.9.9
[2024/11/22 11:43:06] [ info] [ctraces ] version=0.5.7
[2024/11/22 11:43:06] [ info] [input:tail:tail.0] initializing
[2024/11/22 11:43:06] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/11/22 11:43:06] [debug] [tail:tail.0] created event channels: read=25 write=26
[2024/11/22 11:43:06] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2024/11/22 11:43:06] [debug] [input:tail:tail.0] inotify watch fd=32
[2024/11/22 11:43:06] [debug] [input:tail:tail.0] scanning path /var/log/longrun/traffic*.log
[2024/11/22 11:43:06] [debug] [input:tail:tail.0] cannot read info from: /var/log/longrun/traffic*.log
[2024/11/22 11:43:06] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/longrun/traffic*.log'
[2024/11/22 11:43:06] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/11/22 11:43:06] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2024/11/22 11:43:06] [ info] [filter:kubernetes:kubernetes.0] token updated
[2024/11/22 11:43:06] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/11/22 11:43:06] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/11/22 11:43:06] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2024/11/22 11:43:06] [debug] [http_client] not using http_proxy for header

• Steps to reproduce the problem:
  Run the Helm command : helm upgrade --install fluent-bit fluent/fluent-bit -f fluentbit-values.yaml

Expected behavior

Fluent-bit should be able to parse the log from the /var/log/longrun/traffic*.log and push to Elasticsearch/stdout

Screenshots

Your Environment

• Version used: 3.2.1
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes 1.29.9
• Server type and version:
• Operating System and version: Ubuntu 20.04.6 LTS
• Filters and plugins: mentioned above

Additional context

i have few applications running, and i am generating traffic on few endpoints of the application and logging the response codes in the traffic-resp.log file.
To parse the log, i am using parser plugin as mentioned above

log file content:

2024-11-22 00:42:25,206 - Endpoint: http://cnn.ipam.com - Status: 200
2024-11-22 00:42:25,219 - Endpoint: https://cnn.ipamssl.com - Error: HTTPSConnectionPool(host='cnn.ipamssl.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
2024-11-22 00:45:03,891 - Endpoint: https://cnn.longrun.com - Status: 200
2024-11-22 00:45:03,932 - Endpoint: https://guestbook.longrun.com - Status: 200
2024-11-22 00:45:03,950 - Endpoint: https://httpbin.longrun.com - Status: 200
2024-11-22 00:45:03,973 - Endpoint: https://guestbook.defaultsnissl.com - Status: 200

[patrick-stephens]

This is almost always down to permissions on the filesystem and how it is mounted into the container. You need to confirm that there are no security restrictions preventing access to the files in the directory and the directory itself (it is possible to be able to list a directory without being able to read files in it for example).

You can use the debug image to then allow you to shell in and check. There are various options in the helm chart to let you modify the mounts to satisfy any additional security requirements you may need.

As a separate check, also make sure it is not mounting dangling symlinks in: https://chronosphere.io/learn/fluent-bit-kubernetes-filter/#Troubleshooting%20misconfiguration%20woes
That's also a mounting issue - I've seen it before with docker to kubelet symlinks.

[raviganapatp]

I got the issue. I was running the script in the kubernetes master server, and the input plugin was looking for the file in filesystem in worker nodes since fluent-bit daemonsets/pods are deployed only on the worker nodes and not on the master. How can we enable the fluentbit pod deployed in master node as well ?

[patrick-stephens]

I got the issue. I was running the script in the kubernetes master server, and the input plugin was looking for the file in filesystem in worker nodes since fluent-bit daemonsets/pods are deployed only on the worker nodes and not on the master. How can we enable the fluentbit pod deployed in master node as well ?

That's probably more a question for the chart but I imagine there's a node selector in there as well you can configure.

[patrick-stephens]

I can't see anything specifically limiting it in the OSS chart: https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit

https://github.com/fluent/helm-charts/blob/89c9dbbc31529f816ae075a8e83ff5e37c7586c1/charts/fluent-bit/values.yaml#L282

If you're not using that chart then you have to contact whoever created it, that's not something we can support.

[raviganapatp]

Thanks for the quick revert!. You can close the issue