Skip to content

Stream HTML test report processing to fix OOM on large reports#922

Merged
bulldozer-bot[bot] merged 10 commits intodevelopfrom
finlayw/streaming
Feb 23, 2026
Merged

Stream HTML test report processing to fix OOM on large reports#922
bulldozer-bot[bot] merged 10 commits intodevelopfrom
finlayw/streaming

Conversation

@FinlayRJW
Copy link
Contributor

@FinlayRJW FinlayRJW commented Feb 23, 2026

Before this PR

HtmlReportPostProcessor loads entire HTML test report files into memory with Files.readString() and processes them with a DOTALL regex over the full content. For tests that produce enormous stdout, this causes OOM in the Gradle daemon.

See memory usage before:
Screenshot 2026-02-23 at 12 39 26

After this PR

==COMMIT_MSG==
HTML test report post-processing now streams line-by-line using a StreamState state machine, bounding memory to O(single line) regardless of file size.
==COMMIT_MSG==

HtmlReportPostProcessor reads and writes one line at a time via BufferedReader/BufferedWriter, writing to a temp file and atomically replacing the original. A StreamState state machine tracks position within the HTML structure:

  • DEFAULT — passes lines through verbatim
  • AWAITING_PRE — after seeing a <h2>standard output/error</h2> header, waits for the <pre> tag
  • IN_PRE — inside a <pre> block, applies log formatting/filtering to each line

See memory usage after:
Screenshot 2026-02-23 at 12 39 56

Also adds an integration test that prints 500K JSON log lines and verifies the post-processor handles it with a constrained 256MB Gradle daemon heap.

The unused Writable interface has been removed.

Possible downsides?

@changelog-app
Copy link

changelog-app bot commented Feb 23, 2026

Generate changelog in changelog/@unreleased

Type (Select exactly one)

  • Feature (Adding new functionality)
  • Improvement (Improving existing functionality)
  • Fix (Fixing an issue with existing functionality)
  • Break (Creating a new major version by breaking public APIs)
  • Deprecation (Removing functionality in a non-breaking way)
  • Migration (Automatically moving data/functionality to a new system)

Description

HTML test report post-processing now streams line-by-line using a StreamState state machine, bounding memory to O(single line) regardless of file size.

Check the box to generate changelog(s)

  • Generate changelog entry

@changelog-app
Copy link

changelog-app bot commented Feb 23, 2026

Successfully generated changelog entry!

Need to regenerate?

Simply interact with the changelog bot comment again to regenerate these entries.


📋Changelog Preview

💡 Improvements

  • HTML test report post-processing now streams line-by-line using a StreamState state machine, bounding memory to O(single line) regardless of file size. (#922)

@FinlayRJW FinlayRJW requested a review from Copilot February 23, 2026 13:23
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors HTML test report post-processing to stream line-by-line instead of loading entire files into memory, preventing OOM errors when processing large test reports with substantial stdout/stderr output.

Changes:

  • Replaced in-memory regex processing with a streaming state machine (StreamState) that processes HTML files line-by-line
  • Added integration test to verify the streaming approach handles 500K log lines within a 256MB heap constraint
  • Removed the unused Writable interface

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
TestReportFormattingPluginIntegrationTest.java Adds integration test for large report handling with constrained heap; updates copyright year
Writable.java Removes unused interface
StreamState.java Introduces streaming state machine for line-by-line HTML processing
HtmlReportPostProcessor.java Refactors from in-memory regex to streaming with temp file approach

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@FinlayRJW FinlayRJW requested a review from CRogers February 23, 2026 16:20
@bulldozer-bot bulldozer-bot bot merged commit adc1f7e into develop Feb 23, 2026
5 checks passed
@bulldozer-bot bulldozer-bot bot deleted the finlayw/streaming branch February 23, 2026 16:49
@autorelease3
Copy link

autorelease3 bot commented Feb 23, 2026

Released 2.28.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants