Guidance on GPG best practices

[jsternberg]

I currently maintain the InfluxData images for chronograf, kapacitor, influxdb, and telegraf. When we release a new version, we go through a process where all of these images get built usually multiple times before a pull request is eventually created.

Unfortunately, we keep running into an issue with the GPG keys and I'm not sure what to do. It usually delays the upload by about a day while I wait for the GPG keyservers to sort themselves out.

We use the following run line in every image to retrieve the GPG key and we will usually get this at least once during our regular build:

RUN set -ex && \
    for key in \
        05CE15085FC09D18E99EFB22684A14CF2582E0C5 ; \
    do \
        gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$key" || \
        gpg --keyserver pgp.mit.edu --recv-keys "$key" || \
        gpg --keyserver keyserver.pgp.com --recv-keys "$key" ; \
    done

+ gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: directory '/root/.gnupg' created
gpg: new configuration file '/root/.gnupg/dirmngr.conf' created
gpg: new configuration file '/root/.gnupg/gpg.conf' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: keyserver receive failed: Address not available
+ gpg --keyserver pgp.mit.edu --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: keyserver receive failed: No data
+ gpg --keyserver keyserver.pgp.com --recv-keys 05CE15085FC09D18E99EFB22684A14CF2582E0C5
gpg: keyserver receive failed: Host is unreachable

Do you have any advice or best practices for GPG keys? Whatever we are doing, it isn't working well for us. I've seen some reports on issues that switching to an ipv4-only GPG keyserver works, but I also found another comment here that says it's a red herring and doesn't really help.

Thanks for any help.

[yosifkit]

TLDR; GPG servers are extremely flaky and we have yet to solve it reliably.

For automated testing, we usually just set our travis builds to repeat multiple times (like php) since it is usually just gpg failures.

For builds on my local machine (which is were most build tests for official image PRs are done) I run an nginx reverse proxy server to man-in-the-middle apt, apk, and gpg requests and cache them.

For the servers that build and push the official images we take an eventually consistent approach and mark builds as "unstable" if one or more tags fail to build and just allow the build to run later. They also have multiple retries for each tag.

Related issues: docker-library/cassandra#131 (comment), docker-library/tomcat#87, docker-library/tomcat#108, docker-library/mysql#263 (comment), docker-library/httpd#66 (comment), docker-library/php#586, docker-library/wordpress#291

[chorrell]

With the node image, we had a lot of issues with the GPG servers, particularly when building under Alpine. We ended up with the following gpg servers which seem to be working well:

; do \
    gpg --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys "$key" || \
    gpg --keyserver hkp://ipv4.pool.sks-keyservers.net --recv-keys "$key" || \
    gpg --keyserver hkp://pgp.mit.edu:80 --recv-keys "$key" ; \
  done

We ended up dropping keyserver.pgp.com entirely.

[chorrell]

For reference:

• GPG failures nodejs/docker-node#591
• Drop keyserver.pgp.com and reorder pgp key servers nodejs/docker-node#632

[yosifkit]

New initiative is to just use one gpg server and then use a proxy for travis builds (and now on the official build servers!). Example here: docker-library/php#666.