Test rule is SIGKILL-ed after timeout instead of waiting for the termination grace period

[dejan-lokar]

Description of the problem / feature request:

Tests that exceed the timeout are sent a SIGKILL signal without waiting for the termination grace period to pass. I observed this for Golang and Kotlin tests, it however works as expected for shell tests.

Feature requests: what underlying problem are you trying to solve with this feature?

Test cleanup.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I reproduced it in https://github.com/dejan-lokar/bazel-termination-grace with a Golang and a Shell test. I run both tests with 3s timeout and an additional 3s for graceful termination. Shell test exits after ~6s as expected, Golang test however exists after ~3s.

bazel test --test_output=streamed --cache_test_results=no --test_timeout=3 --local_termination_grace_seconds=3 //:sh_test
//:sh_test                                                              TIMEOUT in 6.0s

bazel test --test_output=streamed --cache_test_results=no --test_timeout=3 --local_termination_grace_seconds=3 //:go_test
//:go_test                                                              TIMEOUT in 3.0s

What operating system are you running Bazel on?

macOS Big Sur, Version 11.5.2

What's the output of bazel info release?

release 4.2.1

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

git@github.com:dejan-lokar/bazel-termination-grace.git
d21375a86f0179f20d9fd11ac643366c608ff0a5
d21375a86f0179f20d9fd11ac643366c608ff0a5

Have you found anything relevant by searching the web?

This closed issue appears related: #7119. If I add a --experimental_split_xml_generation flag, the Golang test still timeout-s at ~3s.

[dejan-lokar]

I get the same behavior on Ubuntu 20.04.3 LTS.

bazel info release
release 4.2.1

$ bazel test --test_output=streamed --cache_test_results=no --test_timeout=3 --local_termination_grace_seconds=3 //:sh_test
//:sh_test                                                              TIMEOUT in 6.0s

$ bazel test --test_output=streamed --cache_test_results=no --test_timeout=3 --local_termination_grace_seconds=3 //:go_test
//:go_test                                                              TIMEOUT in 3.0s

[lberki]

@comius does the API team feel responsible for testing infrastructure?

[comius]

@comius does the API team feel responsible for testing infrastructure?

We don't have enough knowledge about it at the moment. Perhaps @larsrc-google is a better candidate to triage this?

[larsrc-google]

No, I can't say I know much if anything about the testing infrastructure either.

[larsrc-google]

Did some digging into this. From the Bazel server point of view, the tests are run in exactly the same way. I suspect the cause is some interaction in how the shell handles signals with how tools/test/test-setup.sh sets them up. The command line used to run the tests are very similar:

[output_base]/process-wrapper --timeout=30 --kill_delay=5 --stats=[output_base]/sandbox/processwrapper-sandbox/6/stats.out external/bazel_tools/tools/test/test-setup.sh ./sh_test

and

[output_base]/process-wrapper --timeout=30 --kill_delay=5 --stats=[output_base]/sandbox/processwrapper-sandbox/8/stats.out external/bazel_tools/tools/test/test-setup.sh go_test_/go_test

[dyitzchaki-roku]

Seeing this with a C++ test on release 7.1.1-9fa75dffd135ee81a6459221d1ca1826c259386f