Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: [Snaps E2E] Unified methods and clean up snaps e2e tests #27684

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

bowensanders
Copy link
Contributor

@bowensanders bowensanders commented Oct 7, 2024

Description

Because of several factors, over time, the E2E tests had diverged in their use of different methods to do the same thing. This PR represents work to unify these methods throughout the entire test base. It also was is an attempt to, as well as possible, document each individual step in the tests.

Some delays were removed in lieu of waitForSelector when appropriate.
Other delays were standardized. For example, Firefox requires delays to allow scrollToElement to complete, hence the new Firefox Specific Delay™ has been added to driver.js as delayFirefox(time) because these delays are not reuired in Chrome instances.

Open in GitHub Codespaces

Related issues

Fixes:

Manual testing steps

  1. Watch CI Fly through with ease

Screenshots/Recordings

Before

After

Pre-merge author checklist

Pre-merge reviewer checklist

  • I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed).
  • I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots.

@bowensanders bowensanders added area-snaps e2e-test End to end test for the MetaMask extension team-snaps-platform Snaps Platform team labels Oct 7, 2024
@bowensanders bowensanders self-assigned this Oct 7, 2024
Copy link
Contributor

github-actions bot commented Oct 7, 2024

CLA Signature Action: All authors have signed the CLA. You may need to manually re-run the blocking PR check if it doesn't pass in a few minutes.

@bowensanders
Copy link
Contributor Author

Before there's any question about the added delays, all snaps tests failed test-e2e-firefox until the delays were reinstated.

In an attempt to remain in method parity between all tests, documented delays have been added for all scrollTo actions.

@bowensanders bowensanders marked this pull request as ready for review October 8, 2024 05:06
@bowensanders bowensanders requested a review from a team as a code owner October 8, 2024 05:06
@metamaskbot
Copy link
Collaborator

Builds ready [3bc665b]
Page Load Metrics (1746 ± 42 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint42620341620404194
domContentLoaded1632200717188641
load1641201817468842
domInteractive267346178
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

FrederikBolding
FrederikBolding previously approved these changes Oct 8, 2024
const snapButton1 = await driver.findElement('#connectbip32');
await driver.scrollToElement(snapButton1);

// added delay for firefox (deflake)
await driver.delay(1000);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if it's possible to conditionally add the delay for just firefox?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is, and now is.
Added a function in driver.js called delayFirefox which only delays if firefox is the current browser.

await driver.delay(1000);

// wait for and click connect
await driver.waitForSelector('#connecterrors');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you're doing this in some other tests so I'll just comment once. Why do we need to have both the waitForSelector call here with an increase in the delay?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, because at first, it didn't hurt. In the case now, because those delays are "firefox specific" we need them for chrome instances.

@metamaskbot
Copy link
Collaborator

Builds ready [94f1e6b]
Page Load Metrics (1792 ± 74 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint15412189178915574
domContentLoaded15322093175113967
load15422192179215374
domInteractive15204514019
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@metamaskbot
Copy link
Collaborator

Builds ready [9b116f3]
Page Load Metrics (1796 ± 81 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint29822731719367176
domContentLoaded15782178177415876
load15902269179616981
domInteractive2187472211
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

Mrtenz
Mrtenz previously approved these changes Oct 10, 2024
@bowensanders
Copy link
Contributor Author

Delay: another round of changes i'm working on has to be finished before reviewing this code again.
I hope this to be complete by tuesday 10/15 EOD.

Copy link

sonarcloud bot commented Oct 17, 2024

@metamaskbot
Copy link
Collaborator

Builds ready [dba0b52]
Page Load Metrics (2053 ± 155 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint34628261794663319
domContentLoaded162628052007310149
load170328262053324155
domInteractive19109462311
backgroundConnect5201524521
firstReactRender512911135526
getState494252411
initialActions01000
loadScripts117122341493259124
setupStore1188382010
uiStartup190532062326384184
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

@metamaskbot
Copy link
Collaborator

Builds ready [52b9b24]
Page Load Metrics (2117 ± 83 ms)
PlatformPageMetricMin (ms)Max (ms)Average (ms)StandardDeviation (ms)MarginOfError (ms)
ChromeHomefirstPaint19024111843668321
domContentLoaded18212390207516981
load18382408211717483
domInteractive20192493417
backgroundConnect890422713
firstReactRender482951186330
getState573242210
initialActions01000
loadScripts12981805153314369
setupStore1389292110
uiStartup204528272386219105
Bundle size diffs
  • background: 0 Bytes (0.00%)
  • ui: 0 Bytes (0.00%)
  • common: 0 Bytes (0.00%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-snaps e2e-test End to end test for the MetaMask extension team-snaps-platform Snaps Platform team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants