Skip to content

Comments

chore(tests): Manually print failed test if xcbeautify fails to parse#7493

Merged
itaybre merged 1 commit intomainfrom
itay/print_failures
Feb 20, 2026
Merged

chore(tests): Manually print failed test if xcbeautify fails to parse#7493
itaybre merged 1 commit intomainfrom
itay/print_failures

Conversation

@itaybre
Copy link
Contributor

@itaybre itaybre commented Feb 20, 2026

📜 Description

Adds a fallback step that detects when xcbeautify silently misses failed tests and prints them directly from the raw log, ensuring failures are always visible in CI output:

=== Failing tests from raw output ===
Failing tests:
	SentrySDKInternalTests.testLogger_TransactionWithBindToScope_LogsInsideDispatchQueueMainAsync_simulatesCOCOA1119()

💡 Motivation and Context

When a test process crashes, xcodebuild may not emit the structured output that xcbeautify needs to detect and report failures. In those cases, xcbeautify continues with no failed tests reported (even though CI correctly fails with exit code 65) making it very hard to identify what went wrong without digging into raw logs manually.

This was observed in this CI run.

The fallback works by checking whether xcbeautify reported any failures, and if not, parsing the raw log directly to surface them.

💚 How did you test it?

📝 Checklist

You have to check all boxes before merging:

  • I added tests to verify the changes.
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • Review from the native team if needed.
  • No breaking change or entry added to the changelog.
  • No breaking change for hybrid SDKs or communicated to hybrid SDKs.

Closes #7494
#skip-changelog

@github-actions
Copy link
Contributor

github-actions bot commented Feb 20, 2026

Semver Impact of This PR

🟢 Patch (bug fixes)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).


This PR will not appear in the changelog.


🤖 This preview updates automatically when you update the PR.

@itaybre itaybre added the ready-to-merge Use this label to trigger all PR workflows label Feb 20, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a fallback mechanism to detect and report failed tests when xcbeautify fails to parse test crash output. When a test process crashes, xcodebuild may emit exit code 65, but xcbeautify might not detect the failure to add it to the JUnit XML report. This causes the test report step to succeed (finding no failures) even though the actual test execution failed. The new step detects this discrepancy and extracts failure information directly from the raw xcodebuild log.

Changes:

  • Added step IDs to test execution steps to enable outcome checking in conditional logic
  • Added a new "Analyze raw test output for crashes" step that runs when test execution fails but the test report finds no failures
  • The fallback step parses raw-test-output.log to extract and display failing test names

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@itaybre
Copy link
Contributor Author

itaybre commented Feb 20, 2026

@sentry review

@codecov
Copy link

codecov bot commented Feb 20, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.299%. Comparing base (b66e455) to head (a1287d2).
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##              main     #7493       +/-   ##
=============================================
- Coverage   85.306%   85.299%   -0.007%     
=============================================
  Files          479       479               
  Lines        28604     28604               
  Branches     12416     12418        +2     
=============================================
- Hits         24401     24399        -2     
- Misses        4158      4160        +2     
  Partials        45        45               

see 4 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b66e455...a1287d2. Read the comment docs.

@itaybre itaybre merged commit 6de4555 into main Feb 20, 2026
206 of 223 checks passed
@itaybre itaybre deleted the itay/print_failures branch February 20, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-code-assisted ready-to-merge Use this label to trigger all PR workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(tests): Manually print failed test if xcbeautify fails to parse

2 participants