Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Moved to RMSE for image comparison to account for slight variations in golden image tests #19658

Merged
merged 7 commits into from
Jul 13, 2020
Merged

Moved to RMSE for image comparison to account for slight variations in golden image tests #19658

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
5aab325
Moved to RMSE for image comparison to account for slight variations in
gaaclarke Jul 10, 2020
5eb302f
ran formatter
gaaclarke Jul 10, 2020
d15ad8d
reverted accidental commit
gaaclarke Jul 10, 2020
98797e4
started ignoring transparent pixels and fixed a flakey test
gaaclarke Jul 13, 2020
71a0d0a
stopped ignoring transparent pixels when calculating the rmse
gaaclarke Jul 13, 2020
1e3c8ac
removed print statement
gaaclarke Jul 13, 2020
7811b9b
tweaked the test for transparency
gaaclarke Jul 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenImage.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
#import <XCTest/XCTest.h>
#include <sys/sysctl.h>

static const double kRmseThreshold = 0.5;
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how did you determinate this value?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the tests on my machine without changes and that was the smallest threshold that that contains all the errors, plus it is a reasonably low error.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would be an example of a false positive?

If you change https://github.com/flutter/engine/blob/master/testing/scenario_app/lib/src/platform_view.dart#L523 to 151 or one of the translates by 1 pixel, does the test pass?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't pass when i change it to 151. In that case the RMSE goes from 0.49 to 0.94.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RMSE factors in how big of a diff there is and is relative to the size of the images. So small discrepancies on large images make less of a difference.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM


@interface GoldenImage ()

@end
Expand Down Expand Up @@ -67,8 +69,24 @@ - (BOOL)compareGoldenToImage:(UIImage*)image {
CGContextDrawImage(contextB, CGRectMake(0, 0, widthA, heightA), imageRefB);
CGContextRelease(contextB);

BOOL isSame = memcmp(rawA.mutableBytes, rawB.mutableBytes, size) == 0;
return isSame;
const char* apos = rawA.mutableBytes;
const char* bpos = rawB.mutableBytes;
double sum = 0.0;
for (size_t i = 0; i < size; ++i, ++apos, ++bpos) {
// Skip transparent pixels.
if (*apos == 0 && *bpos == 0 && i % 4 == 0) {
i += 3;
apos += 3;
bpos += 3;
} else {
double aval = *apos;
double bval = *bpos;
double diff = aval - bval;
sum += diff * diff;
}
}
double rmse = sqrt(sum / size);
return rmse <= kRmseThreshold;
}

NS_INLINE NSString* _platformName() {
Expand Down
6 changes: 1 addition & 5 deletions testing/scenario_app/ios/Scenarios/ScenariosUITests/UnobstructedPlatformViewTests.m
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,12 +243,8 @@ - (void)testPlatformViewsMaxOverlays {

XCUIElement* overlay = app.otherElements[@"platform_view[0].overlay[0]"];
XCTAssertTrue(overlay.exists);
XCTAssertEqual(overlay.frame.origin.x, 75);
XCTAssertEqual(overlay.frame.origin.y, 85);
XCTAssertEqual(overlay.frame.size.width, 150);
XCTAssertEqual(overlay.frame.size.height, 190);

XCTAssertFalse(app.otherElements[@"platform_view[0].overlay[1]"].exists);
XCTAssertTrue(CGRectContainsRect(platform_view.frame, overlay.frame));
}

@end