`system_config.workers` value is wrong when validating config on launching

[daipom]

Describe the bug

system_config.workers value is wrong when validating config on launching.

When launching the supervisor process, Fluentd validates the config as follows.
(It may be difficult to understand what this code is doing. This just validates the config before launching each worker)

fluentd/lib/fluent/supervisor.rb

Line 731 in e89092c

Fluent::Engine.run_configure(@conf, dry_run: dry_run)

This calls configure() of each plugin, and tries to find any config error.

However, system_config.workers value is wrong when we use the multiple workers setting in system directive.

Interestingly, in the case <worker 0-N>, the value is correct.

• This caused a bug about out_file: Avoid performing multi-worker check in configure() #3942

A detailed code survey can be found here.

This can cause bugs in config validation in the future.
We should fix this.

• Note: The problem is similar to The reload configuration with the worker tag failed #3469. We may consider fixing it at the same time.

To Reproduce

Use a plugin to print out system_config.workers in configure().

def configure(conf)
  super
  p system_config.workers
end

I fixed out_stdout to reproduce.

--- a/lib/fluent/plugin/out_stdout.rb
+++ b/lib/fluent/plugin/out_stdout.rb
@@ -51,6 +51,8 @@ module Fluent::Plugin
 
       super
 
+      p system_config.workers
+
       @formatter = formatter_create
     end

Then you can see the value is 1, although you set workers 2 in system directive.

Expected behavior

The actual number of workers is printed out.

Your Environment

- Fluentd version: 1.15.3
- Operating system: "Ubuntu 20.04.5 LTS"
- Kernel version: 5.15.0-58-generic

Your Configuration

<system>
  workers 2
</system>

<source>
  @type sample
  tag test
</source>

<match test.**>
  @type stdout
</match>

Your Error Log

2023-02-10 18:47:59 +0900 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-02-10 18:47:59 +0900 [info]: parsing config file is succeeded path="../work/fluent.conf"
2023-02-10 18:47:59 +0900 [info]: gem 'fluentd' version '1.15.3'
1 <--- WRONG!!
2023-02-10 18:47:59 +0900 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [warn]: both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [info]: using configuration file: <ROOT>
  <system>
    workers 2
  </system>
  <source>
    @type sample
    tag "test"
  </source>
  <match test.**>
    @type stdout
  </match>
</ROOT>
2023-02-10 18:47:59 +0900 [info]: starting fluentd-1.15.3 pid=677703 ruby="2.7.5"
2023-02-10 18:47:59 +0900 [info]: spawn command to main:  cmdline=["/home/daipom/.rbenv/versions/2.7.5/bin/ruby", "-r/home/daipom/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/bundler-2.2.31/lib/bundler/setup", "-Eascii-8bit:ascii-8bit", "/home/daipom/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/bin/fluentd", "-c", "../work/fluent.conf", "--under-supervisor"]
2023-02-10 18:47:59 +0900 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-02-10 18:47:59 +0900 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-02-10 18:47:59 +0900 [info]: #1 init workers logger path=nil rotate_age=nil rotate_size=nil
2023-02-10 18:47:59 +0900 [info]: adding match pattern="test.**" type="stdout"
2
2023-02-10 18:47:59 +0900 [info]: adding source type="sample"
2
2023-02-10 18:47:59 +0900 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [warn]: #0 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [warn]: #1 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [info]: #0 starting fluentd worker pid=677723 ppid=677703 worker=0
2023-02-10 18:47:59 +0900 [warn]: #1 both of Plugin @id and path for <storage> are not specified. Using on-memory store.
2023-02-10 18:47:59 +0900 [info]: #1 starting fluentd worker pid=677724 ppid=677703 worker=1
2023-02-10 18:47:59 +0900 [info]: #0 fluentd worker is now running worker=0
2023-02-10 18:47:59 +0900 [info]: #1 fluentd worker is now running worker=1
2023-02-10 18:48:00.003808793 +0900 test: {"message":"sample"}
2023-02-10 18:48:00.003945149 +0900 test: {"message":"sample"}

Additional context

No response

[daipom]

• This caused a bug about out_file: Avoid performing multi-worker check in configure() #3942

Let me add something about #3942, since it may be difficult to understand.

• The validation at startup had not done all the initialization it was supposed to do, so the validation should have always failed.
• That seemed to be working correctly because of this issue, and we missed it.