Broken nodejs.org webhook since December 17th

[nschonni]

The last deploy in https://nodejs.org/github-webhook.log was for nodejs/nodejs.org@a1196dd
The next commit in this repo was 9df1fc4 but it doesn't look like it should affect the hook so maybe something manual happened.

/cc @Trott

[Trott]

I don't know much about the webhook, although I can learn I suppose? But hopefully someone else on @nodejs/build knows what to do?

[XhmikosR]

May I also add that it might be better to rotate the log more frequently since it gets quite huge very fast?

[phillipj]

Here's a couple of pointers I've learned after working with the @nodejs-github-bot which has a similar GitHub webhook setup:

• github-webhook is a web server that gets installed globally on the server where nodejs.org is hosted
• github-webhook systemd service ensures the above web server is running and restarted it accordingly when needed
• github-webhook.json configures the webhook server to listen for certain GitHub webhook events and execute certain commands
• build-site.sh nodejs gets executed by github-webhook when commits lands on master

Hopefully the above gives some hints about what might be wrong or where to look 🤷‍♂

[rvagg]

weird error messages in the system logs for github-webhook:

Jan 14 14:10:48 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2647]: Emitted 'error' event at:
Jan 14 14:10:48 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2647]:     at lazyFs.open (internal/fs/streams.js:273:12)
Jan 14 14:10:48 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2647]:     at FSReqWrap.oncomplete (fs.js:141:20)
Jan 14 14:11:19 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2956]: events.js:167
Jan 14 14:11:19 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2956]:       throw er; // Unhandled 'error' event
Jan 14 14:11:19 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2956]:       ^
Jan 14 14:11:19 infra-digitalocean-ubuntu1604-x64-1 github-webhook[2956]: Error: ENOENT: no such file or directory, open '/home/nodejs/github-webhook.log'

but the log file does exist; perhaps something to do with a file rotation, although we don't have an automatic one yet.

I've restarted the service but don't have access to nodejs/nodejs.org to trigger a new build to see if it works. Can someone with perms push a new commit or go to the Settings in GitHub and re-trigger a master push event?

[nschonni]

I just merged nodejs/nodejs.org#2872 so it should trigger now

[rvagg]

Mon Jan 06 2020 03:10:57 GMT+0000 (Coordinated Universal Time)
Took 226139 ms
stdout: HEAD is now at b8f965d build(deps-dev): bump nock from 11.7.0 to 11.7.1

looks like it's working again

for the record, this just required a sudo systemctl restart github-webhook on the web server for those (few) with access.. as for the cause, I really don't know.

I'll manually rotate that log not but it does need to be automated by logrotate at some point.

[XhmikosR]

FYI https://nodejs.org/github-webhook.log seems to be empty for me

[rvagg]

I emptied it, push to master and it should fill 🤞

[nschonni]

Merged nodejs/nodejs.org#2875 so it should trigger now

[nschonni]

Looks like it logged the deployment now