Windows Service: Stopping service sometimes can not be completed forever

[daipom]

Describe the bug

On Windows, we can run fluentd as a Windows Service.

(Windows PowerShell with Administrative privilege)

# Register
> fluentd.bat --reg-winsvc i
> fluentd.bat --reg-winsvc-fluentdopt '-c C:/test/td-agent.conf -o C:/test/td-agent.log'

# Start the service
> Start-Service fluentdwinsvc

# Stop the service
> Stop-Service fluentdwinsvc

# Unregister
> fluentd.bat --reg-winsvc u

With certain conditions, Stop-Service can not be completed forever.
This happens when multiple supervisor processes are running simultaneously for some reason (perhaps accidentally).

Fluentd uses Event Objects for signals on Windows.

• Windows Service process: https://github.com/fluent/fluentd/blob/v1.15.2/lib/fluent/winsvc.rb#L65-L70
• Supervisor process: https://github.com/fluent/fluentd/blob/v1.15.2/lib/fluent/supervisor.rb#L242

This event's type is Auto-reset event, so the event is automatically unsignalled when one process receives it.

• Win32::Event.new: https://github.com/chef/win32-event/blob/main/lib/win32/event.rb#L63

If another supervisor unintentionally receives the event, the process that should be stopped fails to receive it and continues to run.

This makes the service wait for the stopping process forever.

• https://github.com/fluent/fluentd/blob/v1.15.2/lib/fluent/winsvc.rb#L68

If we kill all fluentd processes (the service process, supervisor, and workers), we can stop the service forcibly.

To Reproduce

1. Register fluentd as a Windows Service.

> fluentd.bat --reg-winsvc i
> fluentd.bat --reg-winsvc-fluentdopt '-c C:/test/td-agent.conf -o C:/test/td-agent.log'

2. We can see 3 ruby processes with a single worker setting.
   • Service process, supervisor, and worker

> Start-Service fluentdwinsvc
> ps -ProcessName ruby
Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
226      22    51128      54896       2.09   1792   0 ruby
128      12    15796      16384       0.25   2812   0 ruby
183      17    38084      43380       1.53   3232   0 ruby

4. We can stop the service while leaving the supervisor and worker alive as following.

# First, make sure the service is stopped.
> Stop-Service fluentdwinsvc

# Then, start the service and immediately stop it.
> Start-Service fluentdwinsvc
> Stop-Service fluentdwinsvc

# We can see 2 ruby processes still alive. (supervisor and worker process)
> ps -ProcessName ruby
Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
  ...                     ...                   ...        ruby
  ...                     ...                   ...        ruby

5. Start the service again, and now we can see 5 processes.
   • This is the situation where multiple supervisor processes are running simultaneously.

> Start-Service fluentdwinsvc
> ps -ProcessName ruby
Handles  NPM(K)    PM(K)      WS(K)     CPU(s)     Id  SI ProcessName
-------  ------    -----      -----     ------     --  -- -----------
  ...                     ...                   ...        ruby
  ...                     ...                   ...        ruby
  ...                     ...                   ...        ruby
  ...                     ...                   ...        ruby
  ...                     ...                   ...        ruby

6. Then we can reproduce this issue by stopping the service again.

Sorry for the Japanese language environment.

> Stop-Service fluentdwinsvc
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
(repeats forever)

The messages may be as follows in English

> Stop-Service
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
(repeats forever)

Expected behavior

The supervisor process of the service stops for sure.

• the pid: https://github.com/fluent/fluentd/blob/v1.15.2/lib/fluent/winsvc.rb#L59

Or, if fluentd doesn't expect multiple supervisors to run simultaneously, all supervisor processes should stop when stopping the service.

Your Environment

- Fluentd version: 1.15.2
- Operating system: Windows 10

Your Configuration

The config has nothing to do with this issue.

<source>
  @type dummy
  tag test.log
  rate 1
  dummy {"message": "This is the test message"}
</source>

<match test.**>
  @type stdout
</match>

Your Error Log

Sorry for the Japanese language environment

> Stop-Service
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
警告: サービス 'Fluentd Windows Service (fluentdwinsvc)' の停止を待っています...
(repeats forever)

The messages may be as follows in English

> Stop-Service
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
WARNING: Waiting for service 'Fluentd Windows Service (fluentdwinsvc)' to stop...
(repeats forever)

Additional context

No response