Skip to content

azure win10 vs2019 machines have been timing out in debugger tests #3014

Description

@joyeecheung

See nodejs/reliability#354 for reference, it seems the earliest CI run that exhibited this was https://ci.nodejs.org/job/node-test-binary-windows-js-suites/16283/, which is for a PR that hasn't landed rebased on top of a doc commit nodejs/node@dcc9589 so it might be infra issues.

Activity

  1. joyeecheung commented on Aug 22, 2022

    @joyeecheung
    MemberAuthor

    The daily master build on the commit prior to the doc commit was also green on those machines: nodejs/node@5e57d24

  2. joyeecheung commented on Aug 22, 2022

    @joyeecheung
    MemberAuthor

    I do not know how to log into those machines, but I suspect that there might be some stale Node.js processes running there that block the port 9229. That could cause these tests to timeout.

  3. richardlau commented on Aug 22, 2022

    @richardlau
    Member

    I couldn't see any stale Node.js processes still running on test-azure_msft-win10_vs2019-x64-1 or test-azure_msft-win10_vs2019-x64-2 but I've just scheduled a reboot of both machines.

  4. joyeecheung commented on Aug 22, 2022

    @joyeecheung
    MemberAuthor

    The failures still showed up with nodejs/node#44342, so there must be something else that is causing the time out. Also I noticed that this only happes with test-azure_msft-win10_vs2019-x64-1 and -2, the tests don't fail on -3.

  5. joyeecheung commented on Aug 22, 2022

    @joyeecheung
    MemberAuthor

    Looking at the build history, this seems to happen more with test-azure_msft-win10_vs2019-x64-1, and somewhat less frequently with -2. -3 and -4 seem to be immune to that.

  6. richardlau commented on Aug 22, 2022

    @richardlau
    Member

    FWIW -1 and -2 did get rebooted (#3014 (comment)) so whatever the problem is it has persisted, or recreated itself, across the reboot.

  7. joyeecheung commented on Aug 22, 2022

    @joyeecheung
    MemberAuthor

    Another observation: it seems sometimes these failures are accompanied by a few random failures from other tests that involve timeouts, e.g. sequential/test-heap-prof, parallel/test-child-process-exec-timeout, see nodejs/reliability#354

  8. tniessen commented on Aug 23, 2022

    @tniessen
    Member

    It's been failing almost every time, making it difficult to land PRs:

    Windows build trend showing lots of failures

  9. joyeecheung commented on Aug 29, 2022

    @joyeecheung
    MemberAuthor

    Looks like the flakes are now gone, the only recent repros in the reliability issues came from resumed builds that weren't rebased. I'll close this issue.

  10. tniessen commented on Aug 29, 2022

    @tniessen
    Member

    Thank you for working on this @joyeecheung :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions