Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

output to redis failing #4000

Open
ricosega opened this issue Jan 4, 2023 · 1 comment
Open

output to redis failing #4000

ricosega opened this issue Jan 4, 2023 · 1 comment
Labels
waiting-for-triage

Comments

@ricosega
Copy link

ricosega commented Jan 4, 2023

Describe the bug

trying to deploy official helm chart in Kubernetes and always getting the same error.
#0 unexpected error error_class=ArgumentError error="unknown keyword: :thread_safe"

To Reproduce

created an image from "fluent/fluentd-kubernetes-daemonset:v1.15-debian-elasticsearch7-1" and just added the package ubuntu-dev-tools in order to be able to install the plugins hiredis and fluentd-plugin-redis-store.

the config from the values is exactly the same as in the default values.yaml but changing the 04_outputs.conf to:

<label @OUTPUT>
  <match **>
    @type redis_store
    @id redis_out
    driver hiredis
    key logstash
    host "#{ENV['REDIS_HOST']}"
    port "#{ENV['REDIS_PORT']}"
    timeout "#{ENV['REDIS_TIMEOUT']}"
    password "#{ENV['REDIS_PASSWORD']}"
    store_type list
  </match>
</label>

Expected behavior

send outputs to redis

Your Environment

- Fluentd version: fluentd 1.15.3
- Operating system: "Debian GNU/Linux 11 (bullseye)"
- Kernel version: 5.10.0-20-amd64

Your Configuration

<label @FLUENT_LOG>
    <match **>
      @type null
      @id ignore_fluent_logs
    </match>
  </label>
  <source>
    @type tail
    @id in_tail_container_logs
    @label @KUBERNETES
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    emit_unmatched_lines true
    <parse>
      @type "multi_format"
      unmatched_lines true
      <pattern>
        format json
        time_key "time"
        time_type string
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        keep_time_key false
      </pattern>
      <pattern>
        format regexp
        expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        keep_time_key false
      </pattern>
    </parse>
  </source>
  <label @KUBERNETES>
    <match kubernetes.var.log.containers.fluentd**>
      @type relabel
      @label @FLUENT_LOG
    </match>
    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
      skip_labels false
      skip_container_metadata false
      skip_namespace_metadata true
      skip_master_url true
    </filter>
    <match **>
      @type relabel
      @label @DISPATCH
    </match>
  </label>
  <label @DISPATCH>
    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_status_num_records_total
        type counter
        desc The total number of incoming records
        <labels>
          tag ${tag}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>
    <match **>
      @type relabel
      @label @OUTPUT
    </match>
  </label>
  <label @OUTPUT>
    <match **>
      @type redis_store
      @id redis_out
      key "logstash"
      host ""
      port 
      timeout 
      password ""
      store_type "list"
    </match>
  </label>
</ROOT>


### Your Error Log

```shell
2023-01-04 22:27:38 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-04 22:27:38 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/../../../etc/fluent/fluent.conf"
2023-01-04 22:27:38 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.14'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.1.5'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.13.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-parser-cri' version '0.1.1'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-redis-store' version '0.2.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-01-04 22:27:38 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-01-04 22:27:38 +0000 [warn]: [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-01-04 22:27:38 +0000 [info]: using configuration file: <ROOT>
  <label @FLUENT_LOG>
    <match **>
      @type null
      @id ignore_fluent_logs
    </match>
  </label>
  <source>
    @type tail
    @id in_tail_container_logs
    @label @KUBERNETES
    path "/var/log/containers/*.log"
    pos_file "/var/log/fluentd-containers.log.pos"
    tag "kubernetes.*"
    read_from_head true
    emit_unmatched_lines true
    <parse>
      @type "multi_format"
      unmatched_lines true
      <pattern>
        format json
        time_key "time"
        time_type string
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        keep_time_key false
      </pattern>
      <pattern>
        format regexp
        expression /^(?<time>.+) (?<stream>stdout|stderr)( (.))? (?<log>.*)$/
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        keep_time_key false
      </pattern>
    </parse>
  </source>
  <label @KUBERNETES>
    <match kubernetes.var.log.containers.fluentd**>
      @type relabel
      @label @FLUENT_LOG
    </match>
    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
      skip_labels false
      skip_container_metadata false
      skip_namespace_metadata true
      skip_master_url true
    </filter>
    <match **>
      @type relabel
      @label @DISPATCH
    </match>
  </label>
  <label @DISPATCH>
    <filter **>
      @type prometheus
      <metric>
        name fluentd_input_status_num_records_total
        type counter
        desc The total number of incoming records
        <labels>
          tag ${tag}
          hostname ${hostname}
        </labels>
      </metric>
    </filter>
    <match **>
      @type relabel
      @label @OUTPUT
    </match>
  </label>
  <label @OUTPUT>
    <match **>
      @type redis_store
      @id redis_out
      key "logstash"
      host ""
      port 
      timeout 
      password ""
      store_type "list"
    </match>
  </label>
</ROOT>
2023-01-04 22:27:38 +0000 [info]: starting fluentd-1.15.3 pid=2118 ruby="3.1.2"
2023-01-04 22:27:38 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd", "-c", "/fluentd/etc/../../../etc/fluent/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "-r", "/fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-elasticsearch-5.1.5/lib/fluent/plugin/elasticsearch_simple_sniffer.rb", "--under-supervisor"]
2023-01-04 22:27:38 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-04 22:27:39 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-01-04 22:27:39 +0000 [info]: adding match in @FLUENT_LOG pattern="**" type="null"
2023-01-04 22:27:39 +0000 [info]: adding match in @KUBERNETES pattern="kubernetes.var.log.containers.fluentd**" type="relabel"
2023-01-04 22:27:39 +0000 [info]: adding filter in @KUBERNETES pattern="kubernetes.**" type="kubernetes_metadata"
2023-01-04 22:27:39 +0000 [warn]: #0 [filter_kube_metadata] !! The environment variable 'K8S_NODE_NAME' is not set to the node name which can affect the API server and watch efficiency !!
2023-01-04 22:27:39 +0000 [info]: adding match in @KUBERNETES pattern="**" type="relabel"
2023-01-04 22:27:39 +0000 [info]: adding filter in @DISPATCH pattern="**" type="prometheus"
2023-01-04 22:27:39 +0000 [info]: adding match in @DISPATCH pattern="**" type="relabel"
2023-01-04 22:27:39 +0000 [info]: adding match in @OUTPUT pattern="**" type="redis_store"
2023-01-04 22:27:39 +0000 [info]: adding source type="tail"
2023-01-04 22:27:39 +0000 [info]: #0 starting fluentd worker pid=2129 ppid=2118 worker=0
2023-01-04 22:27:39 +0000 [error]: #0 unexpected error error_class=ArgumentError error="unknown keyword: :thread_safe"
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-client-0.11.2/lib/redis_client/config.rb:21:in `initialize'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-client-0.11.2/lib/redis_client/config.rb:176:in `initialize'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-client-0.11.2/lib/redis_client.rb:138:in `new'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-client-0.11.2/lib/redis_client.rb:138:in `config'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-5.0.5/lib/redis/client.rb:23:in `config'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-5.0.5/lib/redis.rb:157:in `initialize_client'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/redis-5.0.5/lib/redis.rb:73:in `initialize'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-redis-store-0.2.0/lib/fluent/plugin/out_redis_store.rb:60:in `new'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-redis-store-0.2.0/lib/fluent/plugin/out_redis_store.rb:60:in `start'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:203:in `block in start'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:182:in `block (2 levels) in lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:121:in `block (2 levels) in lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `each'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:120:in `block in lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `each'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/agent.rb:113:in `lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:181:in `block in lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `each'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/root_agent.rb:202:in `start'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:248:in `start'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/engine.rb:147:in `run'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:783:in `block in run_worker'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:1056:in `main_process'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:774:in `run_worker'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:370:in `<top (required)>'
  2023-01-04 22:27:39 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-01-04 22:27:39 +0000 [error]: #0 <internal:/usr/local/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `load'
  2023-01-04 22:27:39 +0000 [error]: #0 /fluentd/vendor/bundle/ruby/3.1.0/bin/fluentd:25:in `<main>'
2023-01-04 22:27:39 +0000 [error]: #0 unexpected error error_class=ArgumentError error="unknown keyword: :thread_safe"
  2023-01-04 22:27:39 +0000 [error]: #0 suppressed same stacktrace



### Additional context

_No response_
@github-actions
Copy link

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

@github-actions github-actions bot added the stale label Apr 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
waiting-for-triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@daipom @ricosega