Enable Windows Arm64 tests in CI

[niyas-sait]

Node CI had a pipeline for testing Windows Arm64 platform sometimes back but it got disabled as some tests were failing. See #2540 (comment)

Linaro has been looking at reviving the CI job and now we have a new standalone job ( https://ci.nodejs.org/job/windows-arm-simple/ ) that is running with all tests passing.

As discussed during the build meet-up in #2994, Re-enabling the CI again would be crucial to avoid regressions and to officially support the Windows Arm64 platform as requested in #2540

We will need to re-enable the Windows Arm64 test stage to https://ci.nodejs.org/job/node-test-binary-windows-js-suites/

[niyas-sait]

Cc @pbo-linaro, @joaocgreis, @mhdawson, @sxa

Moving the email thread to GitHub issue for visibility and tracking

[pbo-linaro]

Job windows-arm-simple is now set to run daily, to evaluate stability in time.
For now, build for arm64 is already done here, but not testing.

[niyas-sait]

@pbo-linaro Last nightly run was failing https://ci.nodejs.org/job/windows-arm-simple/nodes=win10-vs2019-arm64/39/. Could you have a look ?

[pbo-linaro]

After investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.

Those tests are concerned:

• sequential/test-watch-mode-inspect.mjs
• sequential/test-watch-mode.mjs

[niyas-sait]

After investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.

Those tests are concerned:

• sequential/test-watch-mode-inspect.mjs
• sequential/test-watch-mode.mjs

Thanks, @pbo-linaro ! Is it possible to fix the tests to not depend on a particular order?

[pbo-linaro]

I'm trying to contact author of those commits to see if that is normal, and what we can do with it.

[sxa]

Have you got a link to the relevant commits/PRs for reference - is it the ones that added those tests? And do they have the same problem on win-x64?

The bug theme of this year in multiple projects I've been on has been sorting/ordering problems!

[pbo-linaro]

PR:

• test: split watch mode inspector tests to sequential node#44551
• test: deflake watch mode tests node#44621

@MoLow (author). Do you know if something could create trouble with this?

On my machine, I can make some failure appear by increasing the charge of machine, so something clearly has problem with async dependencies.
Why does it appear much more on Windows on Arm machine? Honestly, I don't know, because they are slower, or the presence of fast/slow cores, that create some specific race conditions.

[pbo-linaro]

@sxa I suppose win-x64 CI did not fail with this, else those tests should already been marked as flaky. On our WoA machines, we reproduced that with x64 binaries (instead of arm64), so it's not architecture/code specific.

[richardlau]

After investigation, I can confirm it is flaky tests, randomly failing because of wrong order in output expected.

Those tests are concerned:

• sequential/test-watch-mode-inspect.mjs
• sequential/test-watch-mode.mjs

nodejs/node#44898?

[pbo-linaro]

Thanks @richardlau, I missed that one, but that's exactly the issue.

[aduh95]

Not sure if it's the correct thread to report this, but I'm getting insufficient disk space to complete link on https://ci.nodejs.org/job/node-compile-windows/47458/nodes=win-vs2019-arm64/, which blocks nodejs/node#44250 from landing.

[pbo-linaro]

@aduh95 It is not built on an arm64 machine, but on this agent: test-rackspace-win2012r2_vs2019-x64-1

you can probably open another issue, or simply ask on nodejs slack build channel, if you have access to it.

[joaocgreis]

ARM64 Windows was never enabled by default in the main node-test-* jobs. It has been there as an option but has to be explicitly selected.

I'm not sure we can enable it by default for all runs with the current resources we have now. The machines we have might be enough, but I expect many jobs will queue up when CI is under load. Also, if the machines fail, it would be good to have someone around who knows how to disable only ARM64, without disabling the whole Windows job.

I suggest we try enabling it, but be ready to disable it quickly if it starts causing issues. We only need to edit the main fanned job and change the default value of RUN_ARM64_TESTS to be set by default.

If tests are failing on master, at some point the procedure was to mark them flaky in https://github.com/nodejs/node/blob/main/test/sequential/sequential.status until a fix lands.

I tried running CI for v18.11.0 with ARM64 enabled, but it didn't run any of the relevant test-binary configurations. For example for https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/: I see RUN_ARM64_TESTS is true in https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/parameters/ but it is considered false in https://ci.nodejs.org/job/node-test-binary-windows-js-suites/17311/console . I checked the safeParameters list and it was not there, so I added it. I believe Jenkins needs to be restarted to update that variable, I'll do that over the weekend if it doesn't happen before. Anyway, unless that variable was recently updated, the cause might be anything else.

[pbo-linaro]

That would be very nice to start running them.
We can try for 24 hours, and see if jobs queue gets larger and larger.

For flaky tests reported here, I created this PR nodejs/node#45049. Different platforms reported problems, so I assume it was good to mark them flaky for all of them.