website build broken during Docker image preparation

[richardlau]

@RafaelGSS has pointed out from the webhook log that the build did not succeed:

stderr: Previous HEAD position was 58704a9... chore: updated issue templates descriptions
stderr: HEAD is now at 7a4e29c... blog: add june 2023 security releases (#5434)
stdout: lts: Pulling from library/node
stdout: c98dda1b0e97: Pulling fs layer
stdout: 9a463ac54ed3: Pulling fs layer
stdout: b70766638915: Pulling fs layer
stdout: 4bcc83cd0070: Pulling fs layer
stdout: 217770c726a2: Pulling fs layer
stdout: cec0e089046a: Pulling fs layer
stdout: 6fedcf8a3041: Pulling fs layer
stdout: 471e0a0d1303: Pulling fs layer
stdout: 4bcc83cd0070: Waiting
stdout: 217770c726a2: Waiting
stdout: cec0e089046a: Waiting
stdout: 471e0a0d1303: Waiting
stdout: 9a463ac54ed3: Download complete
stdout: c98dda1b0e97: Verifying Checksum
stdout: c98dda1b0e97: Download complete
stdout: 217770c726a2: Verifying Checksum
stdout: 217770c726a2: Download complete
stdout: b70766638915: Verifying Checksum
stdout: b70766638915: Download complete
stdout: 6fedcf8a3041: Verifying Checksum
stdout: 6fedcf8a3041: Download complete
stdout: 471e0a0d1303: Verifying Checksum
stdout: 471e0a0d1303: Download complete
stdout: cec0e089046a: Verifying Checksum
stdout: cec0e089046a: Download complete
stdout: 4bcc83cd0070: Verifying Checksum
stdout: 4bcc83cd0070: Download complete
stdout: c98dda1b0e97: Pull complete
stdout: 9a463ac54ed3: Pull complete
stdout: b70766638915: Pull complete
stdout: 4bcc83cd0070: Pull complete
stdout: 217770c726a2: Pull complete
stdout: cec0e089046a: Pull complete
stdout: 6fedcf8a3041: Pull complete
stdout: 471e0a0d1303: Pull complete
stdout: Digest: sha256:d244ade7d0b2ad97002102880142c8b9a4b016ea2308af87e9b150c43987b301
stdout: Status: Downloaded newer image for node:lts
stdout: docker.io/library/node:lts
stdout: Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
stdout: Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
stdout: Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
stdout: Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
stdout: Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
stdout: Fetched 9169 kB in 2s (3673 kB/s)
stdout: Reading package lists...
stderr: E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
stderr: E: Sub-process returned an error code
event="push", match="ref == "refs/heads/main" && repository.full_name == "nodejs/nodejs.org"", exec="/home/nodejs/build-site.sh nodejs"
Tue Jun 13 2023 14:15:19 GMT+0000 (Coordinated Universal Time)
Took 46090 ms

The error looks to be from

build/ansible/www-standalone/resources/scripts/build-site.sh

Lines 40 to 57 in e96c917

	docker pull node:lts
	docker run \
	--rm \
	-v ${clonedir}:/website/ \
	-v /home/nodejs/.npm:/npm/ \
	node:lts \
	bash -c " \
	apt-get update && apt-get install -y rsync && \
	addgroup nodejs --gid ${nodeuid} && \
	adduser nodejs --uid ${nodeuid} --gid ${nodegid} --gecos nodejs --disabled-password && \
	su nodejs -c ' \
	npm config set loglevel http && \
	npm config set cache /npm/ && \
	cd /website/ && \
	npm ci && \
	$build_cmd \
	' \
	"

[richardlau]

The failing command is

build/ansible/www-standalone/resources/scripts/build-site.sh

Line 47 in e96c917

apt-get update && apt-get install -y rsync && \

which is failing on the infra machine:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts bash
root@8f4af573f6ce:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
Fetched 9169 kB in 2s (4534 kB/s)
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
root@8f4af573f6ce:/#

node:lts is the currently the same as node:lts-bookworm and fails:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts-bookworm bash
root@843ac074029e:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [17.6 kB]
Fetched 9169 kB in 2s (4786 kB/s)
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code
root@843ac074029e:/#

but node:lts-bullseye succeeds:

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm node:lts-bullseye bash
root@8526f963090e:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB]
Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Get:4 http://deb.debian.org/debian bullseye/main amd64 Packages [8183 kB]
Get:5 http://deb.debian.org/debian-security bullseye-security/main amd64 Packages [245 kB]
Get:6 http://deb.debian.org/debian bullseye-updates/main amd64 Packages [14.8 kB]
Fetched 8651 kB in 2s (4689 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libpopt0
Suggested packages:
  openssh-server
The following NEW packages will be installed:
  libpopt0 rsync
0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
Need to get 446 kB of archives.
After this operation, 987 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bullseye/main amd64 libpopt0 amd64 1.18-2 [49.6 kB]
Get:2 http://deb.debian.org/debian bullseye/main amd64 rsync amd64 3.2.3-4+deb11u1 [396 kB]
Fetched 446 kB in 0s (10.6 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpopt0:amd64.
(Reading database ... 22794 files and directories currently installed.)
Preparing to unpack .../libpopt0_1.18-2_amd64.deb ...
Unpacking libpopt0:amd64 (1.18-2) ...
Selecting previously unselected package rsync.
Preparing to unpack .../rsync_3.2.3-4+deb11u1_amd64.deb ...
Unpacking rsync (3.2.3-4+deb11u1) ...
Setting up libpopt0:amd64 (1.18-2) ...
Setting up rsync (3.2.3-4+deb11u1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.31-13+deb11u6) ...
root@8526f963090e:/#

root@infra-digitalocean-ubuntu1604-x64-1:~# docker images
REPOSITORY   TAG            IMAGE ID       CREATED        SIZE
node         lts            aa392f56f466   12 hours ago   1.09GB
node         lts-bookworm   aa392f56f466   12 hours ago   1.09GB
node         lts-bullseye   78b037dbb659   3 weeks ago    996MB

Oddly I can't reproduce the failures with node:lts/node:lts-bookworm on a separate machine.

[mhdawson]

What are the versions of ubuntu on the passing/failing based machine? Maybe that is related.

[mhdawson]

Just wondering since I see ubuntu1604 in the name of the website machine

[richardlau]

That's because it is still running Ubuntu 16.04.
The other machine I was testing on is running RHEL 8.

[targos]

Maybe this can be fixed by upgrading Docker, but I don't know if recent versions of Docker work on Ubuntu 16.04

[richardlau]

As another datapoint, I tried node:lts-bookworm on test-digitalocean-ubuntu1804-docker-x64-1 and the failing apt update command passes there. That machine is running

Docker version 24.0.1, build 6802122

The RHEL 8 machine I am on is running

Docker version 20.10.12, build e91ed57

while the www server is on

Docker version 20.10.7, build f0df350

[richardlau]

FWIW using --security-opt=seccomp:unconfined to start the container allows the bookworm-based containers to run apt-get update && apt-get install -y rsync successfully. I'd rather keep running the production build with the working bullseye-based container than use that option.

root@infra-digitalocean-ubuntu1604-x64-1:~# docker run -it --rm --security-opt=seccomp:unconfined node:lts bash
root@c946f84a1e5c:/# apt-get update && apt-get install -y rsync
Get:1 http://deb.debian.org/debian bookworm InRelease [147 kB]
Get:2 http://deb.debian.org/debian bookworm-updates InRelease [52.1 kB]
Get:3 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
Get:4 http://deb.debian.org/debian bookworm/main amd64 Packages [8904 kB]
Get:5 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [24.2 kB]
Fetched 9176 kB in 2s (4859 kB/s)
Reading package lists... Done
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libpopt0
Suggested packages:
  openssh-server python3-braceexpand
The following NEW packages will be installed:
  libpopt0 rsync
0 upgraded, 2 newly installed, 0 to remove and 46 not upgraded.
Need to get 461 kB of archives.
After this operation, 1046 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 libpopt0 amd64 1.19+dfsg-1 [43.3 kB]
Get:2 http://deb.debian.org/debian bookworm/main amd64 rsync amd64 3.2.7-1 [417 kB]
Fetched 461 kB in 0s (10.7 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libpopt0:amd64.
(Reading database ... 23269 files and directories currently installed.)
Preparing to unpack .../libpopt0_1.19+dfsg-1_amd64.deb ...
Unpacking libpopt0:amd64 (1.19+dfsg-1) ...
Selecting previously unselected package rsync.
Preparing to unpack .../rsync_3.2.7-1_amd64.deb ...
Unpacking rsync (3.2.7-1) ...
Setting up libpopt0:amd64 (1.19+dfsg-1) ...
Setting up rsync (3.2.7-1) ...
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Processing triggers for libc-bin (2.36-9) ...
root@c946f84a1e5c:/#