Update iOS UITest baseline screenshots for ACES CI environment #33842
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This PR relaxes screenshot comparison strictness in iOS UI tests to address consistent false-negative failures caused by minor rendering differences between CI macOS environments.
Changes:
- Add tolerance parameters to multiple iOS UI tests that rely on
VerifyScreenshot/VerifyScreenshotOrSetException. - Use higher tolerance (8%) for ActionSheet and TabbedPage menu tests with larger observed pixel differences.
- Document the use of tolerance in keyboard-related screenshot helper methods.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue27750.cs | Adds 2% screenshot tolerance to EditorShouldNotMoveToBottom UI test. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue26754.cs | Adds 8% screenshot tolerance to VerifyTabbedPageMenuItemTextColor test. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue25200.cs | Adds 8% screenshot tolerance to ActionSheet verification helper. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue24496.cs | Adds 2% tolerance to picker/entry keyboard positioning screenshot checks. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue23158.cs | Adds 2% tolerance to entry clear button visibility screenshot check. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue22715.cs | Adds 2% tolerance to page scroll behavior screenshot check. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue19500.cs | Adds 2% tolerance to editor scrolling behavior screenshot check. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue17123.cs | Adds 2% tolerance to table view titles screenshot check. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/Issues/Issue12156.cs | Adds 2.5% tolerance to large title display screenshot check. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/EntryFeatureTests.cs | Applies 2% tolerance (with cropping) to entry feature keyboard-related screenshot helpers and documents rationale. |
| src/Controls/tests/TestCases.Shared.Tests/Tests/FeatureMatrix/EditorFeatureTests.cs | Applies 2% tolerance (with cropping) to editor feature keyboard-related screenshot helpers and documents rationale. |
PureWeen
force-pushed
the
fix/flaky-uitests-screenshot-tolerance
branch
from
9e6a84e to
f4c2650
Compare
Member
Author
|
/azp run maui-pr-uitests |
|
Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command. |
PureWeen
changed the title
Member
Author
|
/azp run maui-pr-uitests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
PureWeen
force-pushed
the
fix/flaky-uitests-screenshot-tolerance
branch
from
f4c2650 to
c97b667
Compare
Member
Author
|
/azp run maui-pr-uitests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Member
Author
|
/azp run maui-pr-uitests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
The iOS 18.4 UITests have been failing on CI since the switch to ACES agent pools (PR #33790). The failures are screenshot comparison failures due to minor rendering differences between macOS 15 (Sequoia) and macOS 26 (Tahoe). This updates the baseline screenshots to match the new CI environment. Updated baselines: - ActionSheetWithLongTitleShouldDisplayProperly.png - ActionSheetWithManyActionsShouldDisplayProperly.png - ActionSheetWithSixActionsShouldDisplayProperly.png - ContentPage_HideSoftinput_WithPaddingAndBackground.png - ContentPage_HideSoftinput_WithRTLAndPadding.png - ContentPage_Title_WithPaddingAndHideSoftInput.png - EditorShouldNotMoveToBottom.png - LargeTitleDisplayWorks.png - PageShouldNotScroll.png - PickerNewKeyboardIsAboveKeyboard_Entry7.png - SwitchTrackColorTest.png - ValidateEntryClearButtonVisibilityBehavior.png - ValidateTableViewTitles.png - VerifyTabbedPageMenuItemTextColor.png
PureWeen
force-pushed
the
fix/flaky-uitests-screenshot-tolerance
branch
from
c97b667 to
268972c
Compare
Member
Author
|
/azp run maui-pr-uitests |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Member
Author
|
/azp run maui-pr-uitests, maui-pr-devicetests |
|
Azure Pipelines successfully started running 2 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!
Description
The iOS 18.4 UITests have been consistently failing on CI since the switch to ACES agent pools (PR #33790). The failures are all screenshot comparison failures with small pixel differences (0.7% - 7.7%), not functional failures.
Root Cause Analysis
After investigating builds 1271888 (passing, Jan 29) vs 1274940+ (failing, Jan 31+), the key difference is:
MAUI(BOT-0251.Sequoia.arm64)AcesShared(ACES_VM_SharedPool_Tahoe)The rendering differences are due to minor font anti-aliasing and layout variations between macOS versions.
Solution
Updated the baseline screenshots to match the new ACES CI environment rendering. The new baselines were generated on the ACES pool and downloaded from build 1276073.
Updated Baselines