github-bot broken?

[richardlau]

I noticed that the bot reported starting a CI run in nodejs/node#52973 (comment) but no following comment linking to the Jenkins run nor statuses posted back to the PR. It looks like a Jenkins run was started: https://ci.nodejs.org/job/node-test-pull-request/59205/

[richardlau]

post-build-status-update at first glance looks green, but clicking on the jobs and looking at the "Pipeline steps", e.g. https://ci.nodejs.org/job/post-build-status-update/2261443/flowGraphTable/, shows that the "httpRequest" step failed.

The console output can't find the build log, e.g.

java.io.FileNotFoundException: /var/lib/jenkins/jobs/post-build-status-update/builds/2261443/log (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
...

I'll see if I can find it on the Jenkins server (I think they get compressed there).

[richardlau]

On ci:

root@infra-digitalocean-ubuntu14-x64-1:~# ls -al /var/lib/jenkins/jobs/post-build-status-update/builds/2261443
total 976
drwxr-xr-x   3 jenkins jenkins   4096 May 14 07:56 .
drwxr-xr-x 102 jenkins jenkins 933888 May 14 08:37 ..
-rw-r--r--   1 jenkins jenkins  35675 May 14 07:56 build.xml
-rw-r--r--   1 jenkins jenkins      0 May 14 07:56 changelog1986113076434187246.xml
-rw-r--r--   1 jenkins jenkins   6021 May 14 07:56 log.gz
-rw-r--r--   1 jenkins jenkins    102 May 14 07:56 log-index
drwxr-xr-x   2 jenkins jenkins   4096 May 14 07:56 workflow
root@infra-digitalocean-ubuntu14-x64-1:~#

From the uncompressed log:

[2024-05-14T11:56:22.706Z] Fetching upstream changes from https://github.com/nodejs/build.git
[2024-05-14T11:56:22.706Z]  > git --version # timeout=10
[2024-05-14T11:56:22.711Z]  > git --version # 'git version 2.34.1'
[2024-05-14T11:56:22.711Z]  > git fetch --tags --force --progress -- https://github.com/nodejs/build.git +refs/heads/*:refs/remotes/origin/* # timeout=10
[2024-05-14T11:56:23.047Z]  > git rev-parse origin/main^{commit} # timeout=10
[2024-05-14T11:56:23.258Z]  > git config core.sparsecheckout # timeout=10
[2024-05-14T11:56:23.264Z]  > git read-tree -mu HEAD # timeout=10
[2024-05-14T11:56:23.272Z]  > git checkout -f e2397a4d6638c7c644464be8ae219f8580ee6100 # timeout=10
[2024-05-14T11:56:23.963Z] HttpMethod: POST
[2024-05-14T11:56:23.963Z] URL: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:23.964Z] Content-Type: application/json; charset=UTF-8
[2024-05-14T11:56:23.995Z] Sending request to url: http://github-bot.nodejs.org:3333/node/jenkins/end
[2024-05-14T11:56:24.193Z] Treating class org.apache.http.conn.HttpHostConnectException(Connect to github-bot.nodejs.org:3333 [github-bot.nodejs.org/23.253.100.79] failed: Connection refused) as 408 Request Timeout

[richardlau]

The bot isn't running -- on gh-bot:

root@infra-rackspace-debian8-x64-1:~# systemctl status github-bot
● github-bot.service - github-bot
   Loaded: loaded (/lib/systemd/system/github-bot.service; enabled)
   Active: activating (auto-restart) (Result: exit-code) since Tue 2024-05-14 13:00:13 UTC; 8s ago
  Process: 29335 ExecStart=/usr/bin/node server.js (code=exited, status=1/FAILURE)
 Main PID: 29335 (code=exited, status=1/FAILURE)

May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: code: 'MODULE_NOT_FOUND',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: requireStack: [
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: '/home/iojs/github-bot/server.js'
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: ]
May 14 13:00:13 infra-rackspace-debian8-x64-1 node[29335]: }
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 13:00:13 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.
root@infra-rackspace-debian8-x64-1:~#

[richardlau]

From journalctl

-- Logs begin at Tue 2024-05-14 10:38:04 UTC, end at Tue 2024-05-14 13:02:12 UTC. --
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Stopping github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Starting github-bot...
May 14 10:38:04 infra-rackspace-debian8-x64-1 systemd[1]: Started github-bot.
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: internal/modules/cjs/loader.js:883
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: throw err;
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ^
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Error: Cannot find module 'node:path'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: Require stack:
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: - /home/iojs/github-bot/server.js
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:15)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:725:27)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at require (internal/modules/cjs/helpers.js:88:18)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.<anonymous> (/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js:28:21)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module._compile (internal/modules/cjs/loader.js:1063:30)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.load (internal/modules/cjs/loader.js:928:32)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Function.Module._load (internal/modules/cjs/loader.js:769:14)
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: at Module.require (internal/modules/cjs/loader.js:952:19) {
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: code: 'MODULE_NOT_FOUND',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: requireStack: [
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/path-scurry/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/glob.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/node_modules/glob/dist/commonjs/index.js',
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: '/home/iojs/github-bot/server.js'
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: ]
May 14 10:38:05 infra-rackspace-debian8-x64-1 node[14484]: }
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: github-bot.service: main process exited, code=exited, status=1/FAILURE
May 14 10:38:05 infra-rackspace-debian8-x64-1 systemd[1]: Unit github-bot.service entered failed state.

which is #3214 and the inability to update Node.js there until we update the OS all over again.

[targos]

nodejs/github-bot@c5ba723

Landed 18 hours ago.

[targos]

Can't we update Node.js to 14.21.3 (the last 14.x version), or at least 14.18? isaacs/node-glob@57f5551

[Trott]

Oof, sorry I did this with that glob update. That sure seemed safe.... 🥴

[Trott]

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

[Trott]

If upgrading to Node.js 14.18 or later is trouble, and even if it's not, a quick fix would be to revert the glob update.

Oops, probably not because the bot will still install the latest glob and this was introduced in a patch release. I'll open a PR to pin glob instead, which isn't great, but will fix things if the Node.js upgrade isn't something that can happen quickly.

[Trott]

nodejs/github-bot#511 for a temporary workaround

[richardlau]

FWIW apt update offers no update or newer verson of Node.js 14.