Many centos7_docker_image_updater failures

[Haroon-Khel]

Currently all but CentOS7 ppc64le is failing:

CentOS7 x64 - Fails at the docker push stage, started failing since sep 16. Failed intermittently before that

+ docker push index.docker.io/adoptopenjdk/centos7_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos7_build_image]
...
a6f794a4850b: Pushed
unauthorized: authentication required

CentOS7 aarch64 - Most recent failure is first in a while

TASK [gmake : Download make 4.1 source] ****************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to connect to ftp.gnu.org at port 443: [Errno -2] Name or service not known", "status": -1, "url": "https://ftp.gnu.org/gnu/make/make-4.1.tar.gz"}

PLAY RECAP *********************************************************************
localhost                  : ok=67   changed=35   unreachable=0    failed=1    skipped=73   rescued=0    ignored=1  

Ubuntu16.04 armv7l - Consistently failing at build stage

TASK [Common : Install GCC G++ on supported platforms] *************************
changed: [localhost] => (item=g++-4.8)
ok: [localhost] => (item=gcc-4.8)
failed: [localhost] (item=gcc-7) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-7", "msg": "No package matching 'gcc-7' is available"}
failed: [localhost] (item=g++-7) => {"ansible_loop_var": "item", "changed": false, "item": "g++-7", "msg": "No package matching 'g++-7' is available"}

Alpine3 x64 - Fails intermittently

Removing intermediate container a60e39ba7677
 ---> f8612b022860
Step 4/7 : COPY ../. /infrastructure
COPY failed: forbidden path outside the build context: ../. ()

Alpine3 aarch64 - Recent failure is first in a while

 ---> aed0d76723cf
Step 4/7 : COPY ../. /infrastructure
COPY failed: forbidden path outside the build context: ../. ()

Docker Manifest stage will not run if any of the builds fail. Needs to be moved into each build stage so at least the builds that pass can run docker manifest stage

[Haroon-Khel]

@sxa I assume the Ubuntu 16 armv7 failure will go away once #2736 is passed?

[Haroon-Khel]

I've just merged #2736

[Haroon-Khel]

The Alpine failures are due to my changes here https://github.com/adoptium/infrastructure/pull/2751/files#diff-cdadf2592f693c4816b592c519b913084ed2de4a238be6dbf42f75464798334bL7. I'll revert those specific changes as they were not needed for that pr, since I found that I could pass the SHA as a parameter

[Haroon-Khel]

@sxa gcc-7 and g++7 arent installing onto ubuntu16 armv7.

TASK [Common : Install GCC G++ on supported platforms] *************************
changed: [localhost] => (item=g++-4.8)
ok: [localhost] => (item=gcc-4.8)
failed: [localhost] (item=gcc-7) => {"ansible_loop_var": "item", "changed": false, "item": "gcc-7", "msg": "No package matching 'gcc-7' is available"}
failed: [localhost] (item=g++-7) => {"ansible_loop_var": "item", "changed": false, "item": "g++-7", "msg": "No package matching 'g++-7' is available"}

I believe this is due to not adding the ubuntu toolchain repo onto our armv7 machines

infrastructure/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/Common/tasks/Ubuntu.yml

Line 25 in d0da682

- name: Add the ubuntu toolchain repository to apt for gcc-7 on Ubuntu 16.04

On build-sxa-armv7l-ubuntu1604-pi4, could you test to see if adding this repo allows gcc7 and g++7 to be installed? I'd do it myself but I can't connect to the machine

[sxa]

On build-sxa-armv7l-ubuntu1604-pi4, could you test to see if adding this repo allows gcc7 and g++7 to be installed? I'd do it myself but I can't connect to the machine

The pi4 hasn't been online for a while (It's not plugged in - I'm not even sure if it has an SD card in it just now). We should look at whether we can migrate those to run on the 64-bit hosts as it's REALLY not ideal for them to be running on the ones in my office :-)

[sxa]

We could really do with #2121 being resolved to help with this stuff. I'll look at getting that fixed since it shouldn't be pulling in anything other than g++ now.

[Haroon-Khel]

Once #2780 is in, the git sha will get passed to each of the docker images in this job

[Haroon-Khel]

The centos7 x64 job is still failing intermittently
https://ci.adoptopenjdk.net/job/centos7_docker_image_updater/280/execution/node/178/log/

+ docker push index.docker.io/adoptopenjdk/centos7_build_image:linux-amd64
The push refers to repository [docker.io/adoptopenjdk/centos7_build_image]
b5ab97693f84: Preparing
5242a211eede: Preparing
790d8fa8a846: Preparing
29c2c34641c7: Preparing
df5aab5a2f44: Preparing
f1cc94e120e3: Preparing
56f9ea533b37: Preparing
919a65251e7d: Preparing
174f56854903: Preparing
56f9ea533b37: Waiting
919a65251e7d: Waiting
174f56854903: Waiting
f1cc94e120e3: Waiting
df5aab5a2f44: Pushed
5242a211eede: Pushed
b5ab97693f84: Pushed
790d8fa8a846: Pushed
174f56854903: Layer already exists
f1cc94e120e3: Pushed
56f9ea533b37: Pushed
919a65251e7d: Pushed
unauthorized: authentication required

Need to investigate

[sxa]

unauthorized: authentication required

That failure has been seen before: #2626

[sxa]

Moving to November target as we're still getting regular failures on this, although we got a full run on October 21st

[sxa]

Last four had problems (look to be all different)

[Haroon-Khel]

The centos6 failures are being documented/worked on here #2753

I am removing the centos6 build from the job temporarily here #2978

[sxa]

Related: updater job is not being scheduled: #3083

[sxa]

It's mostly been ok recently, but the latest one has some failures which don't seem directly related to the changes so I'll keep this open.

[sxa]

Related: #3211

[sxa]

This is more reliable now, but we should evaluate the failures we're seeing since it still seems to be about a 50% overall success rate at the moment.

[sxa]

I'm going to close this now as we have not had any failures in the last 9 runs.