Override exit code on process timeout

[julianbrost]

As Icinga first sends a SIGTERM to a check plugin on timeout to allow it to terminate gracefully, this is not really part of the plugin API specification and we cannot assume that plugins will handle this correctly and still exit with an exit code that maps to UNKNOWN. Therefore, once Icinga decides to kill a process, force its exit code to 128 to be sure the state will be UNKNOWN
after a timeout.

Note on Windows: TerminateProcess(m_Process, 3); is used there and the second parameter sets the exit code, so this is fine. There's only a small race condition where Icinga may do this right when the process exits by itself. Then the call to TerminateProcess() may fail because it already exited but Icinga would still write "" to the output. The exit code is intact though.

Test

object CheckCommand "sigterm-exit-1" {
	command = ["sh", "-c", "trap 'printf TERM; exit 1' TERM; printf 'check start'; sleep inf & wait $$!"]
	timeout = 10
}

Before

After

fixes #8931

[N-o-X]

This log message feels wrong because it didn't terminate with the here mentioned exit code. I'm not sure if logging the hard coded exit code is of any use here.

[julianbrost]

Indeed. I pushed another version, now there's a slight discrepancy in logging as the real exit code isn't part of the ProcessResult, but that's already the case for processes terminated by a signal:

[2021-07-27 18:01:38 +0200] notice/Process: PID 150 ('sh' '-c' 'trap 'printf TERM; exit 1' TERM; printf 'check start'; sleep inf & wait $!') terminated with exit code 1 after sending SIGTERM
[2021-07-27 18:01:38 +0200] warning/PluginCheckTask: Check command for object 'agent-a-1' (PID: 150, arguments: 'sh' '-c' 'trap 'printf TERM; exit 1' TERM; printf 'check start'; sleep inf & wait $!') terminated with exit code 128, output: check start<Timeout exceeded.>TERM