Repository navigation
node-test-commit is broken #3959
Description
Activity
/cc @nodejs/build-infra
I'm in a meeting the whole day and cannot investigate this.
@ryanaslett is this something you could take a look at?
I did some preliminary investigation into this and it appears as though all three of our jenkins workspace machines can no longer connect to github.
All the clone urls are git@github.com, and result in
git@github.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.So either the url's changed in the build and they used to be https urls, or something changed on the workspace machines to remove the private keys.
Being looking at the keys/setup with Ryan to try to understand what might have changed, have followed some threads but no answer yet.
Found this issue which looks like it might be the same issue after looking at the error more closely and some googling - #3614
Looks like packages were updated yesterday (by looking at /var/log/dpkg.log) on
test-ibm-ubuntu2204-x64-3were some of the failures were being seen.Possible that it ran/failed so fast that all jobs were running failing there.
Have restarted the jenkins agent and job seems to be running now. Looks ike it was just that that
gitjust failing to run from java running the agent after the system updates like was the case reported earlier in #3614.Restarted the agent on the 3 workspace machines that I see being associated with the node-test-commit job.
This job seems to be running ok - https://ci.nodejs.org/job/node-test-commit/75781/
Will leave this open until we confirm that we see jobs running ok across the different workspace machines.
still seeing the same issue on https://ci.nodejs.org/job/node-test-pull-request/63546/
The issue for
node-test-commitis resolved, but there are still other jobs impacted (node-test-commit-linux and node-test-commit-custom-suites-freestyle (test-worker))Reacted by Jordan HarbandI updated and rebooted all 4 ubuntu2204 hosts
That seems to have fixed it, https://ci.nodejs.org/job/node-test-pull-request/63552/ doesn’t have the issue. Closing now, thanks everyone.
Reacted by Michael Dawson and Jordan Harbandfwiw, https://ci.nodejs.org/job/node-test-pull-request/63557/ still has the same issue.
fwiw, https://ci.nodejs.org/job/node-test-pull-request/63557/ still has the same issue.
No it does not
hmm - maybe i linked to the wrong thing. https://ci.nodejs.org/job/node-test-commit/75799/console ?
I don't see any anywhere in those logs anything related to a connection problem with github.com, AFAICT it's working fine
alrighty, i'll dig into it, maybe it's an actual failure. sorry for the noise :-)
The two Hetzner hosted machines also needed to have their agents restarted. I've just done so. (node-test-commit-v8-linux and node-daily-coverity were still broken this morning prior to restarting the agent.)
FWIW AFAICT there was no connectivity issue with GitHub and this was another case of automatic updates on Ubuntu updating the Java Runtime Environment which breaks being able to run
git/bash(others?) until the Jenkins agent is restarted. The last time this happened it was suggested disabling automatic updates -- I've pulled that discussion into its own issue: #3962I did some preliminary investigation into this and it appears as though all three of our jenkins workspace machines can no longer connect to github.
@ryanaslett @mhdawson FYI we don't store the keys for GitHub access on the machines -- instead we have Jenkins configured to inject a secret into the builds to allow fetches.
It looks like all jobs are failing is less than a second, starting with https://ci.nodejs.org/job/node-test-commit/75761/