Skip to content

Conversation

@BenHenning
Copy link
Collaborator

@BenHenning BenHenning commented Sep 25, 2025

RaspberryPiFoundation/blockly#9384 revealed that one of the Block comment tests was a bit fragile in directly validating a specific node ID. Since node IDs can change if new elements on the page require a generated ID, the test could easily fail with changes to core Blockly.

This PR fixes the issue by using a regex check instead of a direct ID match. The 'comment_textarea' part is probably sufficiently unique for the test to be considered correct (though it is possible for the wrong comment to be focused here, but it seems really unlikely with how simple the test is being set up and, if that actually happened, it's sort of incidentally still correct as there would be some comment that has focus).

This PR also includes a number of attempts to improve test stability (particularly in CI):

  • It introduces a CI-specific command that adds a 30 second Mocha test timeout to provide a lot more time for tests to fail (since CI tends to run on less performant machines than used for development).
  • It introduces a synchronization between the Mocha timeout and the timeout used for WebdriverIO's waitFor* functions. Note that this actually will usually be 60 seconds regardless of the configured Mocha timeout due to the way the hook is set up. However, this is still far better than the 5 second default as at least 1 test seemed to fairly regularly fail against that limit.
  • It adds a few missing 'wait' calls for helpers that are performing side effects which ensures better synchronization when using PAUSE_TIME (and possibly better stability when running in CI, too, though it's unclear of pause() actually does anything when the provided timeout is 0).
  • See fix: Improve node ID check robustness & WebdriverIO test CI stability #745 (comment) attempts to further stabilize OS X tests only to yield that there are actual test failures.

This will be merged into the screen reader experimental branch to unblock the core Blockly PR as well.

#746 has been filed to track the discovered test failures.

@BenHenning BenHenning requested a review from a team as a code owner September 25, 2025 20:30
@BenHenning BenHenning requested review from gonfunko and removed request for a team September 25, 2025 20:30
Copy link
Contributor

@gonfunko gonfunko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM once the lint failure is resolved.

@BenHenning
Copy link
Collaborator Author

LGTM once the lint failure is resolved.

Should be fixed now. Thanks for quick review!

@BenHenning BenHenning enabled auto-merge (squash) September 25, 2025 20:34
@BenHenning BenHenning disabled auto-merge September 25, 2025 20:49
@BenHenning BenHenning changed the title fix: Improve node ID check robustness fix: Improve node ID check robustness & WDIO CI stability Sep 25, 2025
@BenHenning BenHenning changed the title fix: Improve node ID check robustness & WDIO CI stability fix: Improve node ID check robustness & WebdriverIO test CI stability Sep 25, 2025
This adds more waits and ensures that WebdriverIO timeouts synchronize
with Mocha's (for better CI stability).
@BenHenning
Copy link
Collaborator Author

I've seen the following tests consistently fail in CI now, even with the robustness improvements:

  1) Move start tests
       Start moving value blocks:
     Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/runner/work/blockly-keyboard-experimentation/blockly-keyboard-experimentation/main/test/webdriverio/test/dist/move_test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)

  2) Scrolling into view
       Insert scrolls new block into view:

      AssertionError: expected false to be true
      + expected - actual

      -false
      +true
      
      at Context.<anonymous> (file:///Users/runner/work/blockly-keyboard-experimentation/blockly-keyboard-experimentation/main/test/webdriverio/test/dist/scroll_test.js:75:25)
      at Generator.next (<anonymous>)
      at fulfilled (file:///Users/runner/work/blockly-keyboard-experimentation/blockly-keyboard-experimentation/main/test/webdriverio/test/dist/scroll_test.js:9:58)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

This very likely suggests that these are real failures, though oddly they don't seem consistently reproducible in local OS X environments.

@BenHenning
Copy link
Collaborator Author

I attempted to slow the OS X tests way down by forcing a 100 millisecond PAUSE_TIME for them (which would improve reliability if this is a timing issue per the change in this PR that adds more missing waits) in ae92cf5. However, it still had the same failures which strongly suggests a logical issue that's specific to the CI OS X environment for unknown reasons.

@BenHenning
Copy link
Collaborator Author

Worth noting there were 2 new flakes in the latest run:

  2) Value expression move tests
       using geras
         "before each" hook for "moving right":
     Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/runner/work/blockly-keyboard-experimentation/blockly-keyboard-experimentation/main/test/webdriverio/test/dist/move_test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)
  3) Value expression move tests
       using thrasos
         "before each" hook for "moving right":
     Error: Timeout of 10000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/Users/runner/work/blockly-keyboard-experimentation/blockly-keyboard-experimentation/main/test/webdriverio/test/dist/move_test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)

I'm a bit surprised that's happening since there should be a 30 second timeout here, not a 10 second one. Either way I can't dig into this more today so going ahead and merging with the failures.

@BenHenning BenHenning enabled auto-merge (squash) September 25, 2025 22:57
@BenHenning BenHenning disabled auto-merge September 25, 2025 22:57
@BenHenning BenHenning merged commit 0888bc2 into main Sep 25, 2025
7 of 8 checks passed
@BenHenning BenHenning deleted the improve-block-comment-test-assertion-robustness branch September 25, 2025 22:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants