Workers generate many Zombie processes over time

I've noticed that our containerised worker agents build up a number of Zombie processes. Almost all of these are either sleep or ps.

Expected behaviour: no zombie/defunct processes are running after a build completes
Actual behaviour: test (I think) targets leave behind zombie sleep/ps processes
Steps to reproduce:

run any test target (I think) that uses test-setup.sh (I think...)
run a ps aux from inside the container

More details

$ docker inspect nativelink-worker -f '{{.State.StartedAt}}'
2026-02-18T13:04:02.866034335Z

$ docker exec nativelink-worker bash -c 'ps axo stat= | grep 'Z' | wc -l'
100

$ docker exec nativelink-worker bash -c 'ps axo stat=,cmd= | grep "Z" | sort | uniq -c'
      7 Z    [bash] <defunct>
     12 Z    [ps] <defunct>
     81 Z    [sleep] <defunct>

The bash processes could be from the docker execs but I believe the ps and sleep processes are being spawned by the test-setup.sh script that Bazel uses:

$ docker exec nativelink-worker ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY	 STAT START   TIME COMMAND
...
worker     13581  0.0  0.0	0     0 ?        Z    14:52   0:00 [sleep] <defunct>
...
worker     18941  0.0  0.0	0     0 ?        Z    15:12   0:00 [sleep] <defunct>
...

$ docker logs nativelink-worker | grep '14:52'            
  2026-02-18T14:52:43.783944Z  INFO nativelink_worker::running_actions_manager: Executing command, args: ["external/bazel_tools/tools/test/test-setup.sh", "<some-test-target>"]
  2026-02-18T14:52:44.004773Z  INFO nativelink_worker::running_actions_manager: Command complete, args: ["external/bazel_tools/tools/test/test-setup.sh", "<some-test-target>"]

$ docker logs nativelink-worker | grep '15:12'
  2026-02-18T15:12:59.507469Z  INFO nativelink_worker::running_actions_manager: Executing command, args: ["external/bazel_tools/tools/test/test-setup.sh", "<different-test-target>"]
  2026-02-18T15:12:59.718031Z  INFO nativelink_worker::running_actions_manager: Command complete, args: ["external/bazel_tools/tools/test/test-setup.sh", "<different-test-target>"]

We've been seeing this behaviour since at least 0.7.x (possibly even earlier), currently using 0.8.0.

We're running on Bazel 9.0.0.

@swarren12

@swarren12 thank you for opening this issue and welcome back.

We'll look into this one right away.

@swarren12

@swarren12 Does the work in #2241 fix this if you enable namespaces?

I noticed that v1.0.0 was released, so before trying anything else I upgraded our farm to the latest version. Maybe unsurprisingly, the zombie processes are still being generated on that version.

I've then updated some workers to an image based off :sq72ivxkjh44pgkm32w03g09pimyxxbb.

So far, I don't see any Zombies on those workers. I'll keep an eye on it over the next few hours but I'd tentatively say that the problem has been fixed with the recent changes.

Edit: might have spoken too soon. Came back from lunch and it seems like those nodes have also spawned some zombies. :(

So, I just realised that the previous test was using a stale config without the use_namespaces flag set (so it defaulted to false).

I tried to turn it on, but now the worker crashes on start-up:

Error: Error { code: Unavailable, messages: ["Namespaces not supported", "Could not make LocalWorker", "main() function failed"] }

The documentation says that use_namespaces is only supported for Linux, but the worker is running inside a Rocky 8 container so I'm not sure what the problem is?

@swarren12

@swarren12 I am going to try and test on Rocky Linux sometime this weekend or next week when I get some time, but it should definitely work there. I cannot think of any reason why use_namespaces would not work on Rocky but I honestly have not stayed up to date on it myself. I only know that some of our customers and open source community use Rocky Linux.

The documentation says that use_namespaces is only supported for Linux, but the worker is running inside a Rocky 8 container so I'm not sure what the problem is?

In retrospect, the problem there is obvious. We can't do the namespacing fix inside a Docker container, as it's already in a chroot and so the unshare call we do to do the namespacing fails. #2290 makes this case clearer and fixes the docs to clarify this.

We'll need to figure out to fix this another way!

@swarren12

The documentation says that use_namespaces is only supported for Linux, but the worker is running inside a Rocky 8 container so I'm not sure what the problem is?

In retrospect, the problem there is obvious. We can't do the namespacing fix inside a Docker container, as it's already in a chroot and so the unshare call we do to do the namespacing fails. #2290 makes this case clearer and fixes the docs to clarify this.

@swarren12 This isn't quite correct. If you use privileged containers it works just fine. Is that an option for you?

Running with --privileged seems to have helped. I'll keep an eye on the container and see if its still spawning zombies.

I'll close this one as resolved and we can make a note in the docs or re-open if there are continued issues. Thank you for keeping us updated.

Navigation Menu

Repository navigation

Workers generate many Zombie processes over time #2173

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Workers generate many Zombie processes over time #2173

Description

Activity

MarcusSorealheis commented on Feb 19, 2026

palfrey commented on Mar 27, 2026

swarren12 commented on Apr 1, 2026

swarren12 commented on Apr 7, 2026

MarcusSorealheis commented on Apr 9, 2026

palfrey commented on Apr 29, 2026

palfrey commented on Apr 30, 2026

swarren12 commented on May 7, 2026

MarcusSorealheis commented on May 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions