jenkins,ansible: arm cross-compilers as docker containers

[rvagg]

• Puts all of the cross compiler logic into Docker containers
• Introduces 18.04 for Node 14+
• Restricts the 16.04 builders to < 14
• Removes the < 10 builders

WIP

[rvagg]

(I made this PR with the new https://cli.github.com/, not a bad experience, can recommend folks give it at try at least)

[rvagg]

These new containers should be basically the same as the current pair of 16.04 machines plus the one we have in ci-release. It copies that for 18.04 but removes 4.x and older toolchains, so we'd only be using it for gcc-6 and then giving us a platform for possibly experimenting with gcc-8, which I have an initial toolchain for in the rpi-newer-crosstools repo.

The new containers will get labels with their base image type in it, so cross-compiler-ubuntu1604-armv7-gcc-4.9 instead of cross-compiler-armv7-gcc-4.9 for the current cross compilers. That gives us differentiation for 18.04 and 16.04 and also gives us an easy break with the older cross-compilers, we just need to change the labels in the jobs and VersionSelectorScript.

I just need to test it. I'll run some short experiments tomorrow during the lull in activity I experience during my midday of switching over binary-arm to these new labels and pointing to this branch's VersionSelectorScript and see how it goes across all the release lines. I only have this deployed on one of the Docker hosts at the moment, so just one container of each type in CI so far. If successful, and nobody disagrees, I can roll it out across the 4 of them.

[mhdawson]

Rubber stamp LGTM, Although now I see it is WIP so I'll come back and take another look later.

[rvagg]

This is active now in CI and it seems to be working nicely so I thought I'd leave it on and see how it goes for the next day.

Change to node-cross-compile is here:
https://github.com/nodejs/jenkins-config-test/commit/d297acf30cdfa5aedaf4e15e1291e2d668efe9a2#diff-755a012f4f062541f886291db9489ca3

Main diff is the labels:

-        <string>cross-compiler-armv6-gcc-4.8</string>
-        <string>cross-compiler-armv6-gcc-4.9.4</string>
-        <string>cross-compiler-armv7-gcc-4.8</string>
-        <string>cross-compiler-armv7-gcc-4.9.4</string>
-        <string>cross-compiler-armv7-gcc-6</string>
+        <string>cross-compiler-ubuntu1604-armv6-gcc-4.9.4</string>
+        <string>cross-compiler-ubuntu1604-armv7-gcc-4.9.4</string>
+        <string>cross-compiler-ubuntu1604-armv7-gcc-6</string>
+        <string>cross-compiler-ubuntu1804-armv7-gcc-6</string>

You'll notice that I've got rid of gcc-4.8 and gcc-4.9, leaving gcc-4.9.4 as the lowest. That's because that's the version Node 10 uses, the others are for older builders. Which also means we're not using the Raspbian cross toolchain anymore, only rpi-newer-crosstools. Which also simplifies setup. I've reflected that in the latest commit which pulls them out of Dockerfile and cc-selector.sh.

Pretty happy with this so far I think.

[rvagg]

We are filling up the Pi's much quicker than previously though, so we're still getting "offline, disk full" pretty quick across the cluster. I need to do something more permanent about disk space. I'm thinking of putting the bulk of each Pi root onto HDD leaving only ~iojs/build on each for SDD which gives us a bit more room to move while not compromising on speed too much. That's going to take a bit of work though so that won't be done in a hurry.

[rvagg]

Also to be clear, these binaries for master and still being tested on Pi 2 Debian Stretch images at their lowest, even though we're cross compiling on 18.04. The crosstools are configured to use libc 2.24, which is what Stretch has. So, we seem to have the same level of support even with bumping the host OS version.

[rvagg]

I've added a git clean -fdx to a post build step in an attempt to reclaim a bit of space, maybe that'll be enough to keep us active.

[rvagg]

I think the next step I want to take here is to set up a docker host for ci-release with just these 2 builders on it, a 4cpu machine rather than the usual 2cpu ones we usually have there. Then after that I can remove the 3 current cross compilers and the cross-compile role in Ansible and we're just left with this new mechanism.

[rvagg]

Have added a release docker host and am running test builds now. iojs+release is configured with the new tags and uses this branch's VersionSelectorScript for the test. I've removed the old cross-compiler ansible scripts and entried for the 3 cross compiler machines from inventory. They'll also need to be de-provisioned and removed from firewalls when this gets merged.

[rvagg]

• master:
  • https://ci-release.nodejs.org/job/iojs+release/5907
  • 12:57:04 Triggering iojs+release » cross-compiler-ubuntu1804-armv7-gcc-6
  • https://nodejs.org/download/test/v14.0.0-test202004175a4f24e7e1/
• v13.x:
  • https://ci-release.nodejs.org/job/iojs+release/5908
  • 12:57:19 Triggering iojs+release » cross-compiler-ubuntu1604-armv7-gcc-6
  • https://nodejs.org/download/test/v13.13.1-test20200417947ddec091/
• v12.x:
  • https://ci-release.nodejs.org/job/iojs+release/5909
  • 12:57:44 Triggering iojs+release » cross-compiler-ubuntu1604-armv7-gcc-6
  • https://nodejs.org/download/test/v12.16.3-test2020041769aeaba9e7/
• v10.x:
  • https://ci-release.nodejs.org/job/iojs+release/5910
  • 12:58:13 Triggering iojs+release » cross-compiler-ubuntu1604-armv7-gcc-4.9.4
  • 12:58:13 Triggering iojs+release » cross-compiler-ubuntu1604-armv6-gcc-4.9.4
  • https://nodejs.org/download/test/v10.20.2-test20200417ef2df6986b/

I'm calling this good. PTAL folks.

[richardlau]

* `master`:
  
  * https://ci-release.nodejs.org/job/iojs+release/5907
  * `12:57:04 Triggering iojs+release » cross-compiler-ubuntu1804-armv7-gcc-6`
  * https://nodejs.org/download/test/v14.0.0-test202004175a4f24e7e1/

FWIW the AIX build for this job looked stuck (had been running for 7+hrs) so I cancelled it.

[richardlau]

@rvagg It looks like the nightly has failed: https://ci-release.nodejs.org/job/iojs+release/5911/nodes=cross-compiler-ubuntu1804-armv7-gcc-6/console

07:00:26 Makefile:837: *** CUSTOMTAG is not set for DISTTYPE=custom.  Stop.

It looks like it's this bit of the job config which is now being triggered since the cross-compiler has moved into a docker container and is overriding the disttype to custom instead of nightly:

# manually force all docker builds to be "custom" / "experimental", also forced
# in the promotion tool on the server
if [[ "X${OSVARIANT}" == "Xdocker" ]]; then
  disttype=custom
  RELEASE_URLBASE="https://nodejs.org/download/experimental/"
fi

From the job log:

07:00:26 + RELEASE_URLBASE=https://nodejs.org/download/nightly/
07:00:26 + '[' Xnightly == Xtest ']'
07:00:26 + '[' Xnightly == Xv8-canary ']'
07:00:26 + '[' Xnightly == Xrc ']'
07:00:26 + [[ Xdocker == \X\d\o\c\k\e\r ]]
07:00:26 + disttype=custom
07:00:26 + RELEASE_URLBASE=https://nodejs.org/download/experimental/

[rvagg]

hah, I have very vague memories of this, I guess it was me, I just don't recall that "experimental" thing.

I've commented that section out with a note pointing back here. Will watch today's nightlies.

[rvagg]

https://nodejs.org/download/nightly/v14.0.0-nightly202004204af0598134/
looks good I think, it's green at least https://ci-release.nodejs.org/job/iojs+release/5916/

[rvagg]

updated links to VersionSelectorScript in ci/node-cross-compile and ci-release/iojs+release back to master, not this branch, 🤞