Swap ppc64le builds to use dynamic docker images

[gdams]

We have been creating a centos7 docker build image for quite some time.

I have managed to get jenkins to dynamically provision machines using the OSUOSL openstack and have run some test builds:

https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-ppc64le-hotspot/659/
https://ci.adoptopenjdk.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-linux-ppc64le-openj9/719/

My proposal is to switch jdk15 and jdk16 builds to use these machines for a while and then we can consider switching JDK11 and JDK8 too. My reason for holding off is that we will have to eventually delete the static builds machines to free up some resource and I want to prove this out first.

Any objections @sxa??

[aahlenst]

From my POV, it makes sense to use stateless containers for building (less infra problems). But before switching over more builds, we should look at some potential issues:

• If we need to alter the images as part of a PR to openjdk-build, those changes aren't tested because the builds use an image on Docker Hub.
• In my (very limited) experience, container and non-container builds aren't identical (different parts of the build script are executed). So if we switch entire platforms over to container builds, we're very likely going to see bit rot in the build logic.

[gdams]

@aahlenst the docker image gets pulled from the openjdk-infrastructure repo. (not to be confused with the dockerfiles that live in this repo).

These docker images are an identical copy of our build machines today (they have the full ansible-playbook run on them) which means that the build is going to be identical.

[sxa]

You cannot make such "identical" claims regarding docker images as the kernel is not that of a real distribution in most cases

(For that reason I'm very much against against exclusively running testing in docker containers where it's likely more relevant)

Saying that, if this works I'm ok with it going in, but we need to have something documented about the plugins being used and integration points for the docker stuff as I do not believe any of it has been documented yet.

[aahlenst]

@gdams Sorry, my fault, please ignore.

[gdams]

@sxa we've been successfully building JDK15 and JDK16 using the dynamic docker images for some time now. Would you be happy for me to migrate the JDK8 and JDK11 builds over too? It would require me to delete the two build machines that we have today

[sxa]

WHy does it mandating deleting the existing build machines? Since the builds for JDK8/11/15 don't overlap that shouldn't be a requirement as it doesn't need them all in parallel

[gdams]

@sxa I'm happy to make the change without deleting the build machines. But in the near future we will need to follow up with that task.

[sxa]

Yep that's fine. Happy for this change to be made now
(FYI @andrew-m-leonard)

[sxa]

This needs to be paused until adoptium/temurin-build#2244 is understood and resolved as the builds have been been failing repeatedly on JDK15

[karianna]

This needs to be paused until AdoptOpenJDK/openjdk-build#2244 is understood and resolved as the builds have been failing repeatedly on JDK15

Looks like builds are OK again expect for some xlinux builds on PR testers. So safe to proceed?

[sxa]

This needs to be paused until AdoptOpenJDK/openjdk-build#2244 is understood and resolved as the builds have been failing repeatedly on JDK15

Looks like builds are OK again expect for some xlinux builds on PR testers. So safe to proceed?

Not sure what that's based on ... The ppc64le builds are ok again because I have disabled docker builds for ppc64le as per adoptium/temurin-build#2246 therefore rollout of that to other platforms needs to be halted until we've resolved why and been able to reinstate it safely.