Repository navigation
Unknown client name on test-azure-ubuntu2204_container-arm64 nodes on ci.nodejs.org #4434
Description
Activity
Maybe they're supposed to be named ubuntu2404?
Actually it looks like the host is running e.g.
test-azure-ubuntu2404_container-arm64-2andtest-azure-ubuntu2204_container-arm64-2, however there is no Jenkins agent defined fortest-azure-ubuntu2204_container-arm64-2.nodejs@test-azure-ubuntu2404-docker-arm64-2:~$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f207c61df2f3 node-ci:test-azure-rhel8_container-arm64-2 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-rhel8_container-arm64-2 9294014a9989 node-ci:test-azure-ubuntu2404_container-arm64-4 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-ubuntu2404_container-arm64-4 b0fe71491071 node-ci:test-azure-ubuntu2404_container-arm64-2 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-ubuntu2404_container-arm64-2 2b5a126f3877 node-ci:test-azure-ubuntu2204_container-armv7l-2 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-ubuntu2204_container-armv7l-2 0789d5474ffe node-ci:test-azure-ubuntu2204_container-arm64-2 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-ubuntu2204_container-arm64-2 0775cb734bae node-ci:test-azure-debian13_container-armv7l-2 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-debian13_container-armv7l-2 5625a1d1392d node-ci:test-azure-ubuntu2204_container-arm64-4 "/bin/sh -c 'cd /hom…" 3 weeks ago Up 3 weeks node-ci-test-azure-ubuntu2204_container-arm64-4 nodejs@test-azure-ubuntu2404-docker-arm64-2:~$
There's a mismatch between what is defined in Ansible/Jenkins and what is actually running on the machines
ubuntu2404_docker-arm64-2: server_jobs: 2 containers: - { name: 'test-azure-debian13_container-armv7l-2', os: 'debian13_armv7l', secret: '<redacted>' } - { name: 'test-azure-rhel8_container-arm64-2', os: 'rhel8', secret: '<redacted> } - { name: 'test-azure-ubuntu2204_container-armv7l-2', os: 'ubuntu2204_armv7l', secret: '<redacted>' } - { name: 'test-azure-ubuntu2404_container-arm64-2', os: 'ubuntu2404', secret: '<redacted>' } - { name: 'test-azure-ubuntu2404_container-arm64-4', os: 'ubuntu2404', secret: '<redacted>' }
Suggests we should only be running five containers instead of the seven currently running on the machine. My guess would be that we were running Ubuntu 22.04 arm64 containters before and updated them to Ubuntu 24.04 but the Ubuntu 22.04 containers/systemd config were left behind on the system.
I've stopped and cleared up the Ubuntu 22.04 arm64 systemd services on the
test-azure-ubuntu2404-docker-arm64-*Docker hosts.Generalized procedure (as root). Run
systemctl status jenkins-<name_of_container>.serviceat any point to check the status.systemctl stop jenkins-<name_of_container>.service systemctl disable jenkins-<name_of_container>.service rm /lib/systemd/system/jenkins-<name_of_container>.service systemctl reset-failedThere were also systemd service files for
jenkins-test-azure-debian11_container-armv7l-1.servicejenkins-test-azure-debian11_container-armv7l-2.service
but those were already disabled (and not running). I've also removed those (looks like those containers were replaced with Debian 13 armv7l, which are running).
LGTM now - one of my machines is seemingly failing to connect in as well with an
Authorization failurealthough the log isn't telling me which one. I'll leave this open until I've identified it ...One of my machines is seemingly failing to connect
Identified and stopped (there was a duplicate agent running on one of the machines which had an old, invalid secret key). Closing
Suspect this is a mismatch somewhere on the machines but these errors are in the jenkins logs and should be addressed: