Repository navigation
Support large test suites #5
Description
Activity
That trick didn’t work, unfortunately. The parser did not take the summary line and instead found no tests.
Yikes. Thanks for opening the issue @ben-manes. The parser that I used is pretty naive, it was a quick & dirty attempt. I can switch to a streaming parser and that should be able to parse very large XML documents.
There's still an opportunity to exhaust the heap if you had (say) 9M test failures, each with a very long stack trace or something. We should be thoughtful of that as well and put a cap on the number of failures that we collect for showing the output table and truncate beyond that.
As for why your truncation didn't work, I was examining each
testcasesince my understanding is thatfailuresandskippedare optional in the XML. However, rethinking this, I suspect that what they mean is that the failure property is optional and if omitted means that there are no failures. I thought that I had found a reporter that would omit thefailuresproperty, but now I'm suspicious of my memory. I'll go through this again. If every reporter in common usage always shows afailuresproperty on atestsuitewhen there are actually failures, then we don't need to bother withtestcases unless you want more detailed output. So a mode that doesn't show the test cases table would need not parsetestcases at all. This would be a quick & dirty solution that would make your truncation work.I'm experimenting with a matrix myself and it seems to work fine: https://github.com/crazy-max/buildkit/blob/dc1335ba667df6977e9b02440566c77b297632c7/.github/workflows/build.yml#L224-L275
Result: https://github.com/moby/buildkit/actions/runs/2498837572#summary-6895289837
I'm also using another tool called teststat to have other useful info. Might be cool to have it here too.
We're having the same issue with some builds triggering resource exhaustion due to very large matrix. See for example this run: https://github.com/camunda/zeebe/actions/runs/7895500367/job/21547941014
Couple things that could factor: test result files for these tend to be huge (like ~350MB), but generally the number of tests is low-ish (like < 1k).
My build runs ~9M tests per matrix item, which results in very large junit test results. When experimenting, this action failed due heap exhaustion.
I am currently using publish-unit-test-result-action which had this issue. Prior to their fix I used some bash to truncate with,