-
Notifications
You must be signed in to change notification settings - Fork 13
fix: Improve node ID check robustness & WebdriverIO test CI stability #745
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: Improve node ID check robustness & WebdriverIO test CI stability #745
Conversation
gonfunko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM once the lint failure is resolved.
Should be fixed now. Thanks for quick review! |
This adds a 30 second timeout for WDIO tests, though it's not yet enabled on CI.
This adds more waits and ensures that WebdriverIO timeouts synchronize with Mocha's (for better CI stability).
|
I've seen the following tests consistently fail in CI now, even with the robustness improvements: This very likely suggests that these are real failures, though oddly they don't seem consistently reproducible in local OS X environments. |
|
I attempted to slow the OS X tests way down by forcing a 100 millisecond |
This reverts commit ae92cf5.
|
Worth noting there were 2 new flakes in the latest run: I'm a bit surprised that's happening since there should be a 30 second timeout here, not a 10 second one. Either way I can't dig into this more today so going ahead and merging with the failures. |
RaspberryPiFoundation/blockly#9384 revealed that one of the Block comment tests was a bit fragile in directly validating a specific node ID. Since node IDs can change if new elements on the page require a generated ID, the test could easily fail with changes to core Blockly.
This PR fixes the issue by using a regex check instead of a direct ID match. The 'comment_textarea' part is probably sufficiently unique for the test to be considered correct (though it is possible for the wrong comment to be focused here, but it seems really unlikely with how simple the test is being set up and, if that actually happened, it's sort of incidentally still correct as there would be some comment that has focus).
This PR also includes a number of attempts to improve test stability (particularly in CI):
waitFor*functions. Note that this actually will usually be 60 seconds regardless of the configured Mocha timeout due to the way the hook is set up. However, this is still far better than the 5 second default as at least 1 test seemed to fairly regularly fail against that limit.PAUSE_TIME(and possibly better stability when running in CI, too, though it's unclear ofpause()actually does anything when the provided timeout is 0).This will be merged into the screen reader experimental branch to unblock the core Blockly PR as well.
#746 has been filed to track the discovered test failures.