defunct processes in docker containers chewing up resources during builds

[sxa]

While running in dynamically created docker containers there are a lot of processes ending up in defunct state. This seems to be causing a few problems, potentially including:

• Inability to run with strace on the aarch64 boxes using all available cores Enable SBOM strace for jdk21u x64Linux, ppc64leLinux and aarch64Linux #959 (comment)
• riscv64 jobs running on native boards failing hitting gcc resource issues (Thousands of defunct processes in the jobs before it falls over) (Seen previously)
  These are believed to be connected to each other due to the error in jenkins being similar:

13:47:30  g++-10: fatal error: cannot execute 'as': vfork: Resource temporarily unavailable
13:47:30  compilation terminated.
13:47:30  /usr/bin/bash: fork: retry: Resource temporarily unavailable
13:47:30  gmake[3]: *** [lib/CompileJvm.gmk:154: /home/jenkins/workspace/build-scripts/jobs/jdk22u/jdk22u-linux-riscv64-temurin/workspace/build/openjdkbuild/hotspot/variant-server/libjvm/objs/xAbort.o] Error 1
13:47:30  gmake[3]: *** Waiting for unfinished jobs....
13:47:30  g++-10: fatal error: cannot execute '/usr/lib/gcc/riscv64-linux-gnu/10/cc1plus': vfork: Resource temporarily unavailable
13:47:30  compilation terminated.
13:47:30  gmake[3]: *** [lib/CompileJvm.gmk:154: /home/jenkins/workspace/build-scripts/jobs/jdk22u/jdk22u-linux-riscv64-temurin/workspace/build/openjdkbuild/hotspot/variant-server/libjvm/objs/xAddress.o] Error 1
13:48:05  gmake[2]: *** [make/Main.gmk:253: hotspot-server-libs] Error 2

The problems do not occur when the build is run in the build container outside jenkins from an interactive shell.

Sample `ps` output

sxa@bananapif3:~$ ps -ef | grep defun
jenkins   762144  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762227  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762261  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762297  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762380  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762424  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762458  762059  0 10:20 ?        00:00:00 [sh] <defunct>
jenkins   762463  762059  0 10:20 ?        00:00:00 [make-adopt-buil] <defunct>
jenkins   762569  762059  0 10:20 ?        00:00:00 [makejdk-any-pla] <defunct>
jenkins   762642  762059  0 10:20 ?        00:00:00 [dirmngr] <defunct>
jenkins   762643  762059  0 10:20 ?        00:00:00 [dirmngr] <defunct>
jenkins   762647  762059  0 10:20 ?        00:00:00 [gpg-agent] <defunct>
jenkins   765285  762059  0 10:31 ?        00:00:00 [bash] <defunct>
jenkins   767926  762059  0 10:32 ?        00:00:00 [make] <defunct>
jenkins   768163  762059  0 10:32 ?        00:00:00 [sh] <defunct>
jenkins   768164  762059  0 10:32 ?        00:00:00 [make] <defunct>
jenkins   768184  762059  0 10:32 ?        00:00:00 [bash] <defunct>
jenkins   768185  762059  0 10:32 ?        00:00:00 [gmake] <defunct>
jenkins   768242  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768244  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768262  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768264  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768488  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768489  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768500  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768502  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768508  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768509  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768512  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768513  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768523  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768524  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768534  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768535  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768538  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768541  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768547  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768548  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768579  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768580  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768586  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768587  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768593  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768594  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768607  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768608  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768626  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768627  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768641  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768642  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768651  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768652  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768672  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768673  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768690  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768691  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768712  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768713  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768722  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768723  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768747  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768748  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768763  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768764  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768792  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768793  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768796  762059  0 10:32 ?        00:00:00 [bash] <defunct>
jenkins   768811  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768812  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768842  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768843  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768858  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768859  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768870  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768871  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768885  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768886  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768896  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768897  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768919  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768920  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768943  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768944  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768955  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768956  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768984  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   768985  762059  0 10:32 ?        00:00:00 [tee] <defunct>
jenkins   769008  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769009  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769026  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769027  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769058  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769059  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769085  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769087  762059 99 10:33 ?        00:01:33 [javac] <defunct>
jenkins   769088  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769089  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769106  762059  4 10:33 ?        00:00:04 [gmake] <defunct>
jenkins   769120  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769121  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769427  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769428  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769435  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769436  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769443  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769444  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769453  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769454  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769456  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769457  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769473  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769474  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769490  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769491  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769497  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769498  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769500  762059  0 10:33 ?        00:00:00 [g++-10] <defunct>
jenkins   769501  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769502  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769503  762059 79 10:33 ?        00:00:43 [cc1plus] <defunct>
jenkins   769504  762059  0 10:33 ?        00:00:00 [as] <defunct>
jenkins   769513  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769514  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769522  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769525  762059  0 10:33 ?        00:00:00 [g++-10] <defunct>
jenkins   769526  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769527  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769528  762059 78 10:33 ?        00:00:39 [cc1plus] <defunct>
jenkins   769529  762059  0 10:33 ?        00:00:00 [as] <defunct>
jenkins   769539  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769540  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769552  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769553  762059  0 10:33 ?        00:00:00 [bash] <defunct>
jenkins   769555  762059  0 10:33 ?        00:00:00 [g++-10] <defunct>
jenkins   769556  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769557  762059  0 10:33 ?        00:00:00 [tee] <defunct>
jenkins   769558  762059 70 10:33 ?        00:00:28 [cc1plus] <defunct>
jenkins   769559  762059  0 10:33 ?        00:00:00 [as] <defunct>
jenkins   769571  762059  0 10:34 ?        00:00:00 [bash] <defunct>
jenkins   769574  762059  0 10:34 ?        00:00:00 [g++-10] <defunct>
jenkins   769575  762059  0 10:34 ?        00:00:00 [tee] <defunct>
jenkins   769576  762059  0 10:34 ?        00:00:00 [tee] <defunct>
jenkins   769577  762059 60 10:34 ?        00:00:17 [cc1plus] <defunct>
jenkins   769578  762059  0 10:34 ?        00:00:00 [as] <defunct>
jenkins   769587  762059  0 10:34 ?        00:00:00 [bash] <defunct>
jenkins   769590  762059  0 10:34 ?        00:00:00 [g++-10] <defunct>
jenkins   769591  762059  0 10:34 ?        00:00:00 [tee] <defunct>
jenkins   769592  762059  0 10:34 ?        00:00:00 [tee] <defunct>
jenkins   769593  762059 54 10:34 ?        00:00:13 [cc1plus] <defunct>
jenkins   769594  762059  0 10:34 ?        00:00:00 [as] <defunct>
sxa       769662  761884  0 10:34 pts/11   00:00:00 grep defun
sxa@bananapif3:~$

[sxa]

I initially wondered if this was related to the version of bash we're using, as I was unable to replicate it in a container running locally with a later Ubuntu but I tried building the latest (bash 5.2) and linking it to /bin/bash. The problem still occurred when using the patched container via jenkins.

[sxa]

jenkinsci/docker-agent#714 (comment) suggests that by using another process as the PID1 such problems should not occur.

Currently the jenkins plugin is starting the container with cat as the top level process, which is presumably not interested in reaping the zombies e.g.

docker run [...] ubuntu2004_build_image cat

Based on the comments in the above issue I am trialling with the --init option in dockerRunArgs which seeminly uses tini to change pid 1 to be:

1001           1       0  0 09:44 pts/0    00:00:00 /sbin/docker-init -- cat

[sxa]

Testing arm32+strace and a native (on-board) docker build on riscv64:

• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-linux-arm-temurin/354/consoleFull (Noting that teh strace process is dominating the CPU time locking out one CPU core while other build processes are far lower (Often <10% CPU)
• https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk22u/job/jdk22u-linux-riscv64-temurin/43/consoleFull