Stream HTML test report processing to fix OOM on large reports#922
Stream HTML test report processing to fix OOM on large reports#922bulldozer-bot[bot] merged 10 commits intodevelopfrom
Conversation
Generate changelog in
|
✅ Successfully generated changelog entry!Need to regenerate?Simply interact with the changelog bot comment again to regenerate these entries. 📋Changelog Preview💡 Improvements
|
There was a problem hiding this comment.
Pull request overview
This PR refactors HTML test report post-processing to stream line-by-line instead of loading entire files into memory, preventing OOM errors when processing large test reports with substantial stdout/stderr output.
Changes:
- Replaced in-memory regex processing with a streaming state machine (
StreamState) that processes HTML files line-by-line - Added integration test to verify the streaming approach handles 500K log lines within a 256MB heap constraint
- Removed the unused
Writableinterface
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| TestReportFormattingPluginIntegrationTest.java | Adds integration test for large report handling with constrained heap; updates copyright year |
| Writable.java | Removes unused interface |
| StreamState.java | Introduces streaming state machine for line-by-line HTML processing |
| HtmlReportPostProcessor.java | Refactors from in-memory regex to streaming with temp file approach |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
...ain/java/com/palantir/witchcraft/java/logging/gradle/testreport/HtmlReportPostProcessor.java
Show resolved
Hide resolved
...ogging/src/main/java/com/palantir/witchcraft/java/logging/gradle/testreport/StreamState.java
Outdated
Show resolved
Hide resolved
...ogging/src/main/java/com/palantir/witchcraft/java/logging/gradle/testreport/StreamState.java
Outdated
Show resolved
Hide resolved
...ogging/src/main/java/com/palantir/witchcraft/java/logging/gradle/testreport/StreamState.java
Outdated
Show resolved
Hide resolved
...ogging/src/main/java/com/palantir/witchcraft/java/logging/gradle/testreport/StreamState.java
Show resolved
Hide resolved
|
Released 2.28.0 |
Before this PR
HtmlReportPostProcessorloads entire HTML test report files into memory withFiles.readString()and processes them with aDOTALLregex over the full content. For tests that produce enormous stdout, this causes OOM in the Gradle daemon.See memory usage before:

After this PR
==COMMIT_MSG==
HTML test report post-processing now streams line-by-line using a
StreamStatestate machine, bounding memory to O(single line) regardless of file size.==COMMIT_MSG==
HtmlReportPostProcessorreads and writes one line at a time viaBufferedReader/BufferedWriter, writing to a temp file and atomically replacing the original. AStreamStatestate machine tracks position within the HTML structure:<h2>standard output/error</h2>header, waits for the<pre>tag<pre>block, applies log formatting/filtering to each lineSee memory usage after:

Also adds an integration test that prints 500K JSON log lines and verifies the post-processor handles it with a constrained 256MB Gradle daemon heap.
The unused
Writableinterface has been removed.Possible downsides?