Excessive memory usage creating war overlay #596
Description
Affected version
3.5.0
Bug description
Despite 3.5.0 containing plexus archiver with codehaus-plexus/plexus-archiver#390 we still see java.lang.OutOfMemoryError: Java heap space when building a war with overlay.
The resulting heap shows
1,631 instances of "org.apache.commons.compress.archivers.zip.ScatterZipOutputStream" that occupy 270.07 MB (65.37%) (-Xmx512m).
Our build regularly (constantly) fails on CI but passes locally with the same JVM args, so presumably there is something like storage speed that is causing more buffers to be created whn running in CI than locally.
[INFO] --- war:3.5.0:war (default-war) @ client-master ---
[INFO] Packaging webapp
[INFO] Assembling webapp [client-master] in [/jenkins/workspace/builders_URR-pr-builder_PR-21431/products/core-cm/target/core-cm]
[info] Copying manifest...
[INFO] Processing war project
[INFO] Copying webapp webResources [/jenkins/workspace/builders_URR-pr-builder_PR-21431/products/core-cm/target/generated-resources] to [/jenkins/workspace/builders_URR-pr-builder_PR-21431/products/core-cm/target/core-cm]
[INFO] Copying webapp resources [/jenkins/workspace/builders_URR-pr-builder_PR-21431/products/core-cm/src/main/webapp]
[INFO] Processing overlay [ id org.jenkins-ci.main:jenkins-war]
[155.592s][info][gc] GC(40) Garbage Collection (Allocation Rate) 388M(76%)->178M(35%)
[160.249s][info][gc] GC(41) Garbage Collection (Allocation Rate) 222M(43%)->190M(37%)
[INFO] Building war: /jenkins/workspace/builders_URR-pr-builder_PR-21431/products/core-cm/target/core-cm.war
[161.194s][info][gc] GC(42) Garbage Collection (Allocation Rate) 242M(47%)->400M(78%)
[162.997s][info][gc] Relocation Stall (ForkJoinPool-1-worker-2) 3.449ms
[162.997s][info][gc] Relocation Stall (ForkJoinPool-1-worker-3) 2.346ms
[162.997s][info][gc] Allocation Stall (ForkJoinPool-1-worker-3) 107.233ms
[163.043s][info][gc] GC(43) Garbage Collection (Allocation Rate) 400M(78%)->504M(98%)
[163.743s][info][gc] Allocation Stall (ForkJoinPool-1-worker-5) 657.904ms
[163.743s][info][gc] GC(44) Garbage Collection (Allocation Stall) 508M(99%)->508M(99%)
[164.793s][info][gc] Allocation Stall (ForkJoinPool-1-worker-1) 1705.860ms
[164.843s][info][gc] Allocation Stall (ForkJoinPool-1-worker-7) 1694.301ms
[164.844s][info][gc] GC(45) Garbage Collection (Allocation Stall) 508M(99%)->510M(100%)
[164.844s][info][gc] Allocation Stall (ForkJoinPool-1-worker-2) 1656.572ms
[164.844s][info][gc] Allocation Stall (ForkJoinPool-1-worker-3) 1653.882ms
[164.844s][info][gc] Out Of Memory (ForkJoinPool-1-worker-3)
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1850.hprof ...
JVM flags (configured in .mvn/jvm.config) are -Xmx512m -XX:+UseZGC -verbose:gc -XX:+HeapDumpOnOutOfMemoryError -XX:+TieredCompilation -XX:TieredStopAtLevel=1
Apache Maven 3.9.11 (3e54c93a704957b63ee3494413a2b544fd3d825b)
Java version: 21.0.9, vendor: Eclipse Adoptium, runtime: /opt/tools/jdk21
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "6.6.105+", arch: "amd64", family: "unix"
NB when running the default GC (G1) we still encounter the OOM but also see GCLocker issues potentially https://bugs.openjdk.org/browse/JDK-8192647 (hence using ZGC as an attempted workaround).
1.5k of the object references are from a Thread, the others (2 each) from 40+ ForkJoinPool-1-worker-xx Threads.
for the thread the references are split 833/710 between a java.util.concurrent.ConcurrentLinkedDeque$Itr and a org.apache.commons.compress.archivers.zip.ParallelScatterZipCreator