Compilers are too old on many CI machines

[targos]

Found with nodejs/node#30028

Expected: g++ 6.3.0 or clang++ 8.0.0

V8 requires C++14 support, meaning we need at least g++ 5.x everywhere.

node-test-commit-v8-linux

• infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)
• test-nearform_intel-ubuntu1604-x64-1: (CXX=g++, 5.4.0)

node-test-commit-freebsd

• test-digitalocean-freebsd11-x64-1: (CXX=ccache c++, 6.0.0)

node-test-commit-custom-suites-freestyle

• test-rackspace-ubuntu1604-x64-1: (CXX=ccache g++, 5.4.0)

node-test-commit-arm

• test-packetnet-ubuntu1604-arm64-2: (CXX=ccache g++, 5.4.0)

node-test-commit-linux-containered

• test-joyent-ubuntu1804_sharedlibs_container-x64-4: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-10: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-1: (CXX=ccache g++, 5.4.0)
• test-softlayer-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-7: (CXX=ccache g++, 5.4.0)

node-test-commit-linux

• test-rackspace-debian8-x64-1: (CXX=ccache g++, 4.9.2)
• test-rackspace-ubuntu1604-x64-2: (CXX=ccache g++, 5.4.0)

[rvagg]

node-test-commit-v8-linux
infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

That's the benchmark machine, non-vm I think. I don't believe we have a strategy for dealing with that one because upgrades will change benchmark results. Bumping it to 18.04 would be nice. Similar story for test-nearform_intel-ubuntu1604-x64-{1,2} but we have the option of just adding in a new compiler I think. @nodejs/benchmarking, thoughts? Implications of doing a broad OS upgrade on these machines?

node-test-commit-linux-containered

This is odd because they're supposed to be all 18.04 as of last week, but it looks like I made a boo boo:

build/ansible/roles/docker/templates/ubuntu1804_sharedlibs.Dockerfile.j2

Line 1 in f060da4

FROM ubuntu:16.04

I'll get that sorted today.

node-test-commit-freebsd

🤷‍♂ anyone inclined to figure this out? I don't have time to work out how to upgrade this or get Ansible upgraded so my suggestion is to exclude it in https://github.com/nodejs/build/blob/master/jenkins/scripts/VersionSelectorScript.groovy for newer Node until it is sorted out

... the rest ...

The remaining ones are Ubuntu 16.04 and Debian 8. We have a decision point here:

1. Introduce new machines that have newer compilers and maintain new and old, forking depending on the version of Node
2. Get newer compilers onto these machines and use select-compiler.sh to do the work of switching depending on Node version (we've not done this for Debian & Ubuntu before). We can also do a similar job with build labels and scripting in Jenkins itself, but I select-compiler.sh seems to be a more contributor-friendly path.
3. Remove Ubuntu 16.04 and Debian 8 from the supported list for Node 14 (or 13?) so we can drop them in VersionSelectorScript.groovy. This probably isn't a great idea since Ubuntu 16.04 is supported till 2021 and will go into extended (paid) support until 2024. Debian 8 still has until the middle of 2020, but maybe that's early enough to put it into retirement from our testing? Compiler upgrades on Debian are a nightmare.
4. Upgrade compilers (where we can) and stop testing with older compilers for the older release lines. This would mean that, while we still say we support X compiler for Y release line, we just don't fully test that permutation in our infra.

I'm not in favour of adding more machines, that's more overhead for our stretched people-resources. Compiler switching is feasible but still a fair bit of work and adds a lot of complexity that adds to our maintenance burden (and the burden of onboarding new people, which constantly increases as we grow complexity).

[rvagg]

gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1) now on the sharedlibs / containered machines thanks to a proper Ubuntu 18.04 update.

[targos]

node-test-commit-v8-linux
infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

That's the benchmark machine, non-vm I think. I don't believe we have a strategy for dealing with that one because upgrades will change benchmark results. Bumping it to 18.04 would be nice. Similar story for test-nearform_intel-ubuntu1604-x64-{1,2} but we have the option of just adding in a new compiler I think. @nodejs/benchmarking, thoughts? Implications of doing a broad OS upgrade on these machines?

There's no ideal solution. For node-test-commit-v8-linux, we could probably just use other machines. I guess the benchmark one was chosen because it's powerful?
But we're going to have issues anyway if the benchmark machine is not updated. After V8 7.9 lands on master, we won't be able to run benchmarks on that branch.

[mhdawson]

In terms of running on the benchmark machine I think we just did that to avoid affecting other builds. At this point I'd be in favor of building on other machines as well, but you are right that we'd still need to update the benchmark machine as well.

[rvagg]

@mhdawson I've removed infra-softlayer-ubuntu1404-x64-2 from node-test-commit-v8-linux and will apply #1975 to the nearform benchmarking machines. I've also moved the select-compiler.sh script run to just before make for all platforms (instead of being inside of the arch if's) so that when #1975 is merged it should select gcc-6 for master.

[rvagg]

I think we have this sorted from our end:

node-test-commit-v8-linux

• infra-softlayer-ubuntu1404-x64-2: (CXX=g++, 4.9.4)

removed

• test-nearform_intel-ubuntu1604-x64-1: (CXX=g++, 5.4.0)

updated and using gcc-6 for >= 12

node-test-commit-freebsd

• test-digitalocean-freebsd11-x64-1: (CXX=ccache c++, 6.0.0)

#1973 but it doesn't sound like this is critical, clang 6 might be OK, but we can yank this when it breaks. I think we'll be yanking FreeBSD 11 soon anyway.

node-test-commit-custom-suites-freestyle

• test-rackspace-ubuntu1604-x64-1: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >=12

node-test-commit-arm

• test-packetnet-ubuntu1604-arm64-2: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >= 12

node-test-commit-linux-containered

• test-joyent-ubuntu1804_sharedlibs_container-x64-4: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-10: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-1: (CXX=ccache g++, 5.4.0)
• test-softlayer-ubuntu1804_sharedlibs_container-x64-5: (CXX=ccache g++, 5.4.0)
• test-digitalocean-ubuntu1804_sharedlibs_container-x64-7: (CXX=ccache g++, 5.4.0)

properly using Ubuntu 18.04 now and have a newer compiler

node-test-commit-linux

• test-rackspace-debian8-x64-1: (CXX=ccache g++, 4.9.2)

Debian 8 excluded for Node > 12

• test-rackspace-ubuntu1604-x64-2: (CXX=ccache g++, 5.4.0)

Using gcc-6 for >= 12

You have more green now in a new run of your V8 7.9 @ https://ci.nodejs.org/job/node-test-pull-request/26158/, the failures on Linux now are the two Alpine containers that aren't happy.

[targos]

Thanks a lot, Rod!