Repository navigation
Enable Windows Arm64 tests in CI #3046
Description
Activity
Cc @pbo-linaro, @joaocgreis, @mhdawson, @sxa
Moving the email thread to GitHub issue for visibility and tracking
Reacted by Pierrick Bouvier@pbo-linaro Last nightly run was failing https://ci.nodejs.org/job/windows-arm-simple/nodes=win10-vs2019-arm64/39/. Could you have a look ?
Reacted by Pierrick BouvierAfter investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.
Those tests are concerned:
- sequential/test-watch-mode-inspect.mjs
- sequential/test-watch-mode.mjs
After investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.
Those tests are concerned:
- sequential/test-watch-mode-inspect.mjs
- sequential/test-watch-mode.mjs
Thanks, @pbo-linaro ! Is it possible to fix the tests to not depend on a particular order?
I'm trying to contact author of those commits to see if that is normal, and what we can do with it.
Reacted by Niyas SaitHave you got a link to the relevant commits/PRs for reference - is it the ones that added those tests? And do they have the same problem on win-x64?
The bug theme of this year in multiple projects I've been on has been sorting/ordering problems!
PR:
- test: split watch mode inspector tests to sequential node#44551
- test: deflake watch mode tests node#44621
@MoLow (author). Do you know if something could create trouble with this?
On my machine, I can make some failure appear by increasing the charge of machine, so something clearly has problem with async dependencies.
Why does it appear much more on Windows on Arm machine? Honestly, I don't know, because they are slower, or the presence of fast/slow cores, that create some specific race conditions.After investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.
Those tests are concerned:
- sequential/test-watch-mode-inspect.mjs
- sequential/test-watch-mode.mjs
Thanks @richardlau, I missed that one, but that's exactly the issue.
Not sure if it's the correct thread to report this, but I'm getting
insufficient disk space to complete linkon https://ci.nodejs.org/job/node-compile-windows/47458/nodes=win-vs2019-arm64/, which blocks nodejs/node#44250 from landing.ARM64 Windows was never enabled by default in the main
node-test-*jobs. It has been there as an option but has to be explicitly selected.I'm not sure we can enable it by default for all runs with the current resources we have now. The machines we have might be enough, but I expect many jobs will queue up when CI is under load. Also, if the machines fail, it would be good to have someone around who knows how to disable only ARM64, without disabling the whole Windows job.
I suggest we try enabling it, but be ready to disable it quickly if it starts causing issues. We only need to edit the main fanned job and change the default value of RUN_ARM64_TESTS to be set by default.
If tests are failing on master, at some point the procedure was to mark them flaky in https://github.com/nodejs/node/blob/main/test/sequential/sequential.status until a fix lands.
I tried running CI for v18.11.0 with ARM64 enabled, but it didn't run any of the relevant test-binary configurations. For example for https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/: I see RUN_ARM64_TESTS is true in https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/parameters/ but it is considered false in https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/console . I checked the safeParameters list and it was not there, so I added it. I believe Jenkins needs to be restarted to update that variable, I'll do that over the weekend if it doesn't happen before. Anyway, unless that variable was recently updated, the cause might be anything else.
That would be very nice to start running them.
We can try for 24 hours, and see if jobs queue gets larger and larger.For flaky tests reported here, I created this PR nodejs/node#45049. Different platforms reported problems, so I assume it was good to mark them flaky for all of them.
75 remaining items
Thanks, @joaocgreis. nodejs/node#47020 has been landed. Hope we can enable the CI by default now.
Reacted by Pierrick BouvierARM64 Windows tests are running by default 🎉
Will monitor for issues for a few days, this can then be closed after #3250 lands.
Reacted by Niyas SaitExcellent! Great work @joaocgreis and @StefanStojanovic!
- added 3 commits that reference this issue
on Mar 23, 2023 - added a commit that references this issue
on Mar 27, 2023 - added a commit that references this issue
on Mar 28, 2023 - added a commit that references this issue
on Mar 30, 2023 - added a commit that references this issue
on Jul 6, 2023
Node CI had a pipeline for testing Windows Arm64 platform sometimes back but it got disabled as some tests were failing. See #2540 (comment)
Linaro has been looking at reviving the CI job and now we have a new standalone job ( https://ci.nodejs.org/job/windows-arm-simple/ ) that is running with all tests passing.
As discussed during the build meet-up in #2994, Re-enabling the CI again would be crucial to avoid regressions and to officially support the Windows Arm64 platform as requested in #2540
We will need to re-enable the Windows Arm64 test stage to https://ci.nodejs.org/job/node-test-binary-windows-js-suites/