Skip to content

Commit 59f8800

Browse files
Copilotslachiewicz
andauthored
Reduce heap usage in Zip archiver to prevent OutOfMemoryError in CI builds (#390)
* Reduce heap usage threshold from 100MB to 10MB in ConcurrentJarCreator The previous threshold of 100000000 (100MB) divided by number of threads was causing OutOfMemoryError issues when creating zip archives, particularly in environments with limited heap space like CI systems. The ByteArrayOutputStream used internally can grow up to 2x the threshold before switching to disk-based storage, leading to excessive memory consumption. Reducing to 10000000 (10MB) reduces memory pressure while still maintaining reasonable performance. For typical builds with 4 threads, this means: - Before: 25MB per stream (potentially 50MB+ with buffer doubling) - After: 2.5MB per stream (potentially 5MB+ with buffer doubling) This change addresses the heap space errors reported in apache/maven CI builds. Co-authored-by: slachiewicz <6705942+slachiewicz@users.noreply.github.com> * Add maximum buffer size cap to ByteArrayOutputStream Cap individual buffer allocations to 16MB to prevent excessive memory usage. The previous implementation allowed buffers to double indefinitely, which could lead to very large single allocations (32MB, 64MB, 128MB, etc.) that contribute to heap exhaustion. With this cap, when more than 16MB of data needs to be stored, the ByteArrayOutputStream will create multiple 16MB buffers instead of one giant buffer. This spreads the memory allocation across multiple smaller chunks and prevents heap fragmentation issues. Combined with the reduced threshold in ConcurrentJarCreator, this provides defense-in-depth against OutOfMemoryError during zip archive creation. Co-authored-by: slachiewicz <6705942+slachiewicz@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: slachiewicz <6705942+slachiewicz@users.noreply.github.com>
1 parent c099a55 commit 59f8800

File tree

2 files changed

+10
-1
lines changed

2 files changed

+10
-1
lines changed

src/main/java/org/codehaus/plexus/archiver/zip/ByteArrayOutputStream.java

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,12 @@ public class ByteArrayOutputStream extends OutputStream {
5757
*/
5858
private static final byte[] EMPTY_BYTE_ARRAY = new byte[0];
5959

60+
/**
61+
* Maximum size of a single buffer (16MB) to prevent excessive memory allocation.
62+
* When buffers need to grow beyond this size, additional buffers are created instead.
63+
*/
64+
private static final int MAX_BUFFER_SIZE = 16 * 1024 * 1024;
65+
6066
/**
6167
* The list of buffers, which grows and never reduces.
6268
*/
@@ -136,6 +142,9 @@ private void needNewBuffer(final int newcount) {
136142
filledBufferSum += currentBuffer.length;
137143
}
138144

145+
// Cap the buffer size to prevent excessive memory allocation
146+
newBufferSize = Math.min(newBufferSize, MAX_BUFFER_SIZE);
147+
139148
currentBufferIndex++;
140149
currentBuffer = new byte[newBufferSize];
141150
buffers.add(currentBuffer);

src/main/java/org/codehaus/plexus/archiver/zip/ConcurrentJarCreator.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ public ConcurrentJarCreator(int nThreads) throws IOException {
112112
*/
113113
public ConcurrentJarCreator(boolean compressAddedZips, int nThreads) throws IOException {
114114
this.compressAddedZips = compressAddedZips;
115-
ScatterGatherBackingStoreSupplier defaultSupplier = new DeferredSupplier(100000000 / nThreads);
115+
ScatterGatherBackingStoreSupplier defaultSupplier = new DeferredSupplier(10000000 / nThreads);
116116
metaInfDir = createDeferred(defaultSupplier);
117117
manifest = createDeferred(defaultSupplier);
118118
directories = createDeferred(defaultSupplier);

0 commit comments

Comments
 (0)