[CI] Centos docker builds failing during yum install

[tvernum]

I don't know if there's any solution to this, but I didn't want the issue to just get lost in build noise.

We seem to have recurring failures building the docker image on Centos because yum fails to retrieve the mirror list.

Feb 13 https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=centos-7&&immutable&&linux&&docker/14/console

05:06:48 Cannot find a valid baseurl for repo: base/7/x86_64
05:06:48 �[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
05:06:48 12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
05:06:49 
05:06:49 The command '/bin/sh -c yum install -y unzip which' returned a non-zero code: 1
05:06:49 

Feb 11 https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+periodic/30/console

11:12:05 Cannot find a valid baseurl for repo: base/7/x86_64
11:12:05 �[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
11:12:05 14: curl#7 - "Failed to connect to 2604:1580:fe02:2::10: Network is unreachable"
11:12:05 The command '/bin/sh -c yum install -y unzip which' returned a non-zero code: 1
11:12:05 
11:12:05 

Feb 10 https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=openjdk12,ES_RUNTIME_JAVA=zulu11,nodes=immutable&&linux&&docker/233/console

05:07:21 Cannot find a valid baseurl for repo: base/7/x86_64
05:07:21 �[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
05:07:21 12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
05:07:21 The command '/bin/sh -c yum update -y &&     yum install -y nc unzip wget which &&     yum clean all' returned a non-zero code: 1
05:07:21 
05:07:21 > Task
```

[elasticmachine]

Pinging @elastic/es-core-infra

[dliappis]

@tvernum thanks for creating this. Ironically enough it seems that this is happening on CentOS-7 workers attempting to yum install inside the container based on centos:7.

I have been unable to reproduce this using a simple reproduction inside a clean CentOS 7 vagrant box with 20 iterations.

Attempted reproduction script

$ cat Dockerfile
FROM centos:7 AS builder
ENV PATH /usr/share/elasticsearch/bin:$PATH
ENV JAVA_HOME /opt/jdk-11.0.2
RUN curl --retry 8 -s https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_linux-x64_bin.tar.gz | tar -C /opt -zxf -
RUN ln -sf /etc/pki/ca-trust/extracted/java/cacerts /opt/jdk-11.0.2/lib/security/cacerts
RUN yum install -y unzip which

for i in {1..20}; do docker build -t testimage . && echo ">>>> Test $i successful"; docker rmi testimage; done

Spinning up a worker similar to the one used in the failures above I observed that the base OS has certain images already present (incl. centos7) and the above reproduction script was basically a no-op; while this speeds up things, it might be the reason behind some stale things.

[dliappis]

Looked at this again.

It mostly happens on centos-7 workers but I also spotted it happening on a Ubuntu kibana-ci, devops-ci Ubuntu (beats) immutable workers and even on an openSUSE Leap 42.3 non immutable worker, so it doesn't look specifically related to centos-7 workers. The history seems to start on Jan 17 this year; this doesn’t really chronologically correspond to the packer cache script fix pr in https://github.com/elastic/elasticsearch/pull/38023/files.

Had a brainstorming session with @atorok on some ideas.

Alpar pointed out that the failing jobs were for non master branches hence not benefiting from caching. He'll work on a simple PR to extend the caching for older versions.

I looked at our Dockerfile and as an additional action we can:

1. Remove yum install -y unzip which from the stage 0 image -- not needed for some time now.
2. Clean up the yum section in stage 1; we still need yum update -y but I'll add a retry loop with some sleep to help even further.

Hopefully with all the above steps combined we will get rid of the noise.

[benwtrent]

Re-occurred in builds:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+periodic/89/console

02:06:40 Cannot find a valid baseurl for repo: base/7/x86_64
02:06:40 �[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
02:06:40 14: curl#7 - "Failed to connect to 2607:f8f8:700:12::10: Network is unreachable"

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.0+periodic/89/console

04:06:23 Cannot find a valid baseurl for repo: base/7/x86_64
04:06:23 �[0mCould not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
04:06:23 12: Timeout on http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')

[danielmitterdorfer]

@benwtrent can we please add at least the relevant build output here to help investigating? Jenkins links are usually invalid after a few weeks. While it is still possible to get the build output it is much more trouble to go to build-stats and dig for the respective logs.

[benwtrent]

@danielmitterdorfer I updated my comment to include the snippet of the timeout failure. I initially did not add them as it did not provide any more information than what was already included in this issue.

[dliappis]

@atorok this issue seems to persist e.g. in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.7+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=immutable&&linux&&docker/40/console. AFAICS updated base images incl. caching support have been building successfully.

This is a list of failures the past few days:

WDYT, should we be more resilient in all cases by introducing the retry commit: 6f52008?