Multi Process Workers documentation may be outdated: buffer path is not configured. specify 'path' in <buffer>

[GuiTeK]

(this message was edited after @repeatedly comment but the issue is still valid)

Describe the bug

The doc states that when using multi process workers and file buffers, path in the <buffer> section should be omitted in favour of root_dir in the <system> section and @id in the output section.

We tried to do it with a <buffer> section for the out_elasticsearch plugin like so:

<system>
    workers 2
    root_dir /var/lib/td-agent
</system>

<store>
    @type elasticsearch

    # ...

    <buffer tag,time>
        @type file
        @id out_elasticsearch

        # ...
    </buffer>
</store>

and we got the following error when starting Fluentd:

2021-10-21 12:19:53 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="buffer path is not configured. specify 'path' in <buffer>"

EDIT: as @repeatedly said, @id should go in the output configuration and not in the <buffer> configuration. So we tried this as well, and then Fluentd crashes without any error message (even with log_level trace):

2021-10-21 13:58:25 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-10-21 13:58:25 +0000 [info]: gem 'fluentd' version '1.13.3'
2021-10-21 13:58:25 +0000 [trace]: registered output plugin 'relabel'
2021-10-21 13:58:25 +0000 [trace]: registered filter plugin 'record_modifier'
2021-10-21 13:58:25 +0000 [trace]: registered output plugin 'copy'
2021-10-21 13:58:25 +0000 [debug]: adding store type="prometheus"
2021-10-21 13:58:25 +0000 [trace]: registered output plugin 'prometheus'
2021-10-21 13:58:25 +0000 [debug]: adding store type="elasticsearch"
2021-10-21 13:58:25 +0000 [trace]: registered output plugin 'elasticsearch'
2021-10-21 13:58:25 +0000 [trace]: registered buffer plugin 'file'

Finally we tried to put the @id in our <match> block directly (not in <store>), and we get the following:

2021-10-21 13:59:50 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.5'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-kafka' version '0.16.3'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.1'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-s3' version '1.6.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-td' version '1.1.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.4.0'
2021-10-21 13:59:50 +0000 [info]: gem 'fluentd' version '1.13.3'
2021-10-21 13:59:50 +0000 [trace]: registered output plugin 'relabel'
2021-10-21 13:59:50 +0000 [trace]: registered filter plugin 'record_modifier'
2021-10-21 13:59:50 +0000 [trace]: registered output plugin 'copy'
2021-10-21 13:59:50 +0000 [debug]: [my_match] adding store type="prometheus"
2021-10-21 13:59:50 +0000 [trace]: registered output plugin 'prometheus'
2021-10-21 13:59:50 +0000 [debug]: [my_match] adding store type="elasticsearch"
2021-10-21 13:59:50 +0000 [trace]: registered output plugin 'elasticsearch'
2021-10-21 13:59:50 +0000 [trace]: registered buffer plugin 'file'
2021-10-21 13:59:50 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="buffer path is not configured. specify 'path' in <buffer>"
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/buf_file.rb:67:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/output.rb:342:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/event_emitter.rb:73:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-elasticsearch-5.0.5/lib/fluent/plugin/out_elasticsearch.rb:204:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:74:in `block in configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `each'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/multi_output.rb:63:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin/out_copy.rb:39:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/plugin.rb:178:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:132:in `add_match'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:74:in `block in configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `each'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/agent.rb:64:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/label.rb:31:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `block in configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `each'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/root_agent.rb:143:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:105:in `configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/engine.rb:80:in `run_configure'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/supervisor.rb:663:in `run_supervisor'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/lib/fluent/command/fluentd.rb:344:in `<top (required)>'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.13.3/bin/fluentd:15:in `<top (required)>'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/bin/fluentd:23:in `load'
  2021-10-21 13:59:50 +0000 [debug]: /opt/td-agent/bin/fluentd:23:in `<main>'

Finally, we did the following, and not only it worked, but it also automatically created subfolders /var/lib/td-agent/worker0 and /var/lib/td-agent/worker1, which suggests that what the doc states is useless/outdated:

<system>
    workers 2
</system>

<store>
    @type elasticsearch

    # ...

    <buffer tag,time>
        @type file
        path /var/lib/td-agent

        # ...
    </buffer>
</store>

Link to the problematic documentation

https://docs.fluentd.org/deployment/multi-process-workers

Expected explanation

Not sure what is the right combination of settings, but clearly the current doc produces an error.

Additional context

No response

[repeatedly]

@id in the section.

@id should be in output configuration, not <buffer>. Could you re-check your configuration?
Example from docs:

<system>
  workers 2
  root_dir /var/log/fluentd
</system>

<match pattern>
  @type forward
  @id out_fwd
  <buffer>
    @type file
  </buffer>
</match>

[GuiTeK]

Hi @repeatedly,

Thank you for your comment! Indeed you are right.

However, we tried to put it everywhere (<store>, <match>, <buffer>) but we get a different error each time and it doesn't work.

I edited my original post to reflect this.