chore(tests): Manually print failed test if xcbeautify fails to parse#7493
chore(tests): Manually print failed test if xcbeautify fails to parse#7493
Conversation
Semver Impact of This PR🟢 Patch (bug fixes) 📋 Changelog PreviewThis is how your changes will appear in the changelog. This PR will not appear in the changelog. 🤖 This preview updates automatically when you update the PR. |
There was a problem hiding this comment.
Pull request overview
This PR adds a fallback mechanism to detect and report failed tests when xcbeautify fails to parse test crash output. When a test process crashes, xcodebuild may emit exit code 65, but xcbeautify might not detect the failure to add it to the JUnit XML report. This causes the test report step to succeed (finding no failures) even though the actual test execution failed. The new step detects this discrepancy and extracts failure information directly from the raw xcodebuild log.
Changes:
- Added step IDs to test execution steps to enable outcome checking in conditional logic
- Added a new "Analyze raw test output for crashes" step that runs when test execution fails but the test report finds no failures
- The fallback step parses raw-test-output.log to extract and display failing test names
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@sentry review |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7493 +/- ##
=============================================
- Coverage 85.306% 85.299% -0.007%
=============================================
Files 479 479
Lines 28604 28604
Branches 12416 12418 +2
=============================================
- Hits 24401 24399 -2
- Misses 4158 4160 +2
Partials 45 45 see 4 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
|
📜 Description
Adds a fallback step that detects when xcbeautify silently misses failed tests and prints them directly from the raw log, ensuring failures are always visible in CI output:
💡 Motivation and Context
When a test process crashes, xcodebuild may not emit the structured output that xcbeautify needs to detect and report failures. In those cases, xcbeautify continues with no failed tests reported (even though CI correctly fails with exit code 65) making it very hard to identify what went wrong without digging into raw logs manually.
This was observed in this CI run.
The fallback works by checking whether xcbeautify reported any failures, and if not, parsing the raw log directly to surface them.
💚 How did you test it?
📝 Checklist
You have to check all boxes before merging:
sendDefaultPIIis enabled.Closes #7494
#skip-changelog