Fix media types in Accept header for Docker Registry

[jasonrudolph]

#2962 reported an problem with the pull requests that Dependabot was opening for the python:slim Docker image. The python:slim Docker image is a multi-architecture image and publishes a manifest list (also known as a "fat manifest"). With an image that uses a manifest list, the manifest list has a digest, and each architecture-specific image in the list has a digest as well. In #2962 we saw that when a Dockerfile depended on python:slim (which again is a multi-architecture image), Dependabot wrongly created a pull request to update the Dockerfile to use the linux/amd64 version of the image. In other words, instead of referring to the digest of the python:slim manifest list, the Dockerfile was now referring to the digest of the linux/amd64 python:slim manifest.

To fix this problem, #3060 added the Accept header used to indicate that we want to receive manifest lists back from the Docker API. In that PR, our Accept header changed as follows:

- application/vnd.docker.distribution.manifest.v2+json, application/json
+ application/vnd.docker.distribution.manifest.list.v2+json, application/json

This worked well for manifest lists, but we've recently learned that this broke the behavior for images that don't use manifest lists. (According to the Docker docs, most images don't use manifest lists).

Because we're no longer telling the Docker API that we accept the application/vnd.docker.distribution.manifest.v2+json media type, when we ask for the manifest for an image that does not use a list, the Docker API falls back to the old v1 media type: application/vnd.docker.distribution.manifest.v1+prettyjws and we end up with the wrong digest.

To resolve this problem, we instead need to tell the Docker API that we accept multiple media types:

application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/json

With this ☝️ as our Accept header, the Docker API gives us the behavior we need for images that use manifest and for images that use manifest lists. If the image uses a manifest, we get back the manifest. If an image uses a manifest list, we get back the manifest list. 😅

Examples

Let's see this in action. To see how it works for an image that uses a plain ole manifest (non-list), we'll use a jenkins image. To see how it works for an image that uses a manifest list, we'll use a python image.

For starters, we'll pull each of these images using the docker CLI and note the digest that it returns. This is the digest that we want Dependabot to use, which means that it's the digest that we want to get back from the Docker REST API.

Manifest (Non-list): Jenkins

$ docker pull jenkins/jenkins:2.263.4-lts-alpine
2.263.4-lts-alpine: Pulling from jenkins/jenkins
Digest: sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a
Status: Image is up to date for jenkins/jenkins:2.263.4-lts-alpine
docker.io/jenkins/jenkins:2.263.4-lts-alpine

To confirm that this image uses a plain manifest, and not a manifest list, we can check the media type:

$ docker buildx imagetools inspect jenkins/jenkins:2.263.4-lts-alpine
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   ... snip
}

Manifest List: Python

$ docker pull python:slim
slim: Pulling from library/python
Digest: sha256:bf3ec573c0ae0d0c619c3f3e0e9490878432bf7a5c63a643b6c39c9878b51191
Status: Image is up to date for python:slim
docker.io/library/python:slim

To confirm that this image uses a manifest list, we can check the media type:

$ docker buildx imagetools inspect library/python:slim
Name:      docker.io/library/python:slim
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
...

Fetching the digest with the Docker REST API

For these examples, we'll use the docker_registry2 gem, since that's the Docker Registry client that dependabot-core uses.

# Create the client
reg = DockerRegistry2.connect("https://registry.hub.docker.com", { user: "jasonrudolph", password: "<redacted>" })

dependabot-core relies on the #digest method to get the digest for an image manifest. Under the hood, that method uses the Docker API endpoint for pulling an image manifest, so we'll use that API directly for demonstration purposes.

Let's fetch the digest for jenkins:

reg.dohead('/v2/jenkins/jenkins/manifests/2.263.4-lts-alpine').headers
# => {:content_length=>"3248", :content_type=>"application/vnd.docker.distribution.manifest.v2+json", :docker_content_digest=>"sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a", :docker_distribution_api_version=>"registry/2.0", :etag=>"\"sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a\"", :date=>"Wed, 10 Feb 2021 21:12:27 GMT", :strict_transport_security=>"max-age=31536000"}

✅ Note that the docker_content_digest is the same digest we got from docker pull. (Good!) And note that the content type is application/vnd.docker.distribution.manifest.v2+json, indicating a plain (non-list) manifest.

Let's fetch the digest for python:

reg.dohead('/v2/library/python/manifests/slim').headers
# => {:content_length=>"1370", :content_type=>"application/vnd.docker.distribution.manifest.v2+json", :docker_content_digest=>"sha256:c63c1906d81f84abbdc8b5af9e1e94f9860ee3f703c136629d6c322960ee08c5", :docker_distribution_api_version=>"registry/2.0", :etag=>"\"sha256:c63c1906d81f84abbdc8b5af9e1e94f9860ee3f703c136629d6c322960ee08c5\"", :date=>"Wed, 10 Feb 2021 21:19:02 GMT", :strict_transport_security=>"max-age=31536000"}

❌ Note that the docker_content_digest is not the digest we got from docker pull. And note that the content type is application/vnd.docker.distribution.manifest.v2+json, despite the fact that this image uses a manifest list.

Now let's adjust the Accept header

Now let's change the Accept header like so:

- application/vnd.docker.distribution.manifest.v2+json, application/json
+ application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/json

Let's try fetching the digest for python again:

reg.dohead('/v2/library/python/manifests/slim').headers
# => {:content_length=>"1862", :content_type=>"application/vnd.docker.distribution.manifest.list.v2+json", :docker_content_digest=>"sha256:bf3ec573c0ae0d0c619c3f3e0e9490878432bf7a5c63a643b6c39c9878b51191", :docker_distribution_api_version=>"registry/2.0", :etag=>"\"sha256:bf3ec573c0ae0d0c619c3f3e0e9490878432bf7a5c63a643b6c39c9878b51191\"", :date=>"Wed, 10 Feb 2021 21:11:58 GMT", :strict_transport_security=>"max-age=31536000"}

✅ Now, note that the docker_content_digest is the same the digest we got from docker pull. 🕺 And note that the content type is application/vnd.docker.distribution.manifest.list.v2+json, indicating that the image uses a manifest list.

If we repeat the process for jenkins, we still get the correct results that we saw previously:

reg.dohead('/v2/jenkins/jenkins/manifests/2.263.4-lts-alpine').headers
# => {:content_length=>"3248", :content_type=>"application/vnd.docker.distribution.manifest.v2+json", :docker_content_digest=>"sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a", :docker_distribution_api_version=>"registry/2.0", :etag=>"\"sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a\"", :date=>"Wed, 10 Feb 2021 21:12:27 GMT", :strict_transport_security=>"max-age=31536000"}

Put it to use in dependabot-core

To make sure this has the desired effect in dependabot-core, let's try a dry-run with each image. I created a test repo with a Dockerfile for the jenkins image and a Dockerfile for the python image. We'll perform a dry-run on each and verify that dependabot-core proposes an update to the same SHAs that we saw from docker pull above:

$ bin/dry-run.rb docker jasonrudolph/bug-free-system --dir jenkins

± Dockerfile
~~~
1c1
< FROM jenkins/jenkins:2.263.4-lts-alpine@sha256:fa0b0f06383944877c8105ad8e7edcac29f05583cef8b0d26cd8c9586e0cb667
---
> FROM jenkins/jenkins:2.263.4-lts-alpine@sha256:6eec4f88e34bce5c8d016e790006f4d5545105491dd6e9f3096e06cc874fd19a
~~~

± Dockerfile
~~~
1c1
< FROM python:slim@sha256:4d92968b26bb6b7b62d957244de86fc1054f03793577d49e85c00864eb03ca07
---
> FROM python:slim@sha256:bf3ec573c0ae0d0c619c3f3e0e9490878432bf7a5c63a643b6c39c9878b51191
~~~

And indeed, dependabot-core suggests updates to the same SHAs that we saw from docker pull.

Why no tests?

Still reading? Whew! You're a trooper. If you're wondering why this pull request doesn't touch any tests, thanks for caring about our test suite! 🙇 @jurre summarizes this for us in #3060 (comment):

The tests have remained unchanged in this PR, because (unfortunately) all the requests and responses are stubbed out.

[jurre]

🎉 thanks for this amazingly thorough write-up

[jurre]

Going to merge this so we can roll it out later today

[thepwagner]

😍 using an Accept header to get the manifest/list in a single roundtrip is neato!

My earlier thinking was biased the RegistryClient built into docker, which doesn't treat the manifest list itself as a thing: https://github.com/docker/cli/blob/70a00157f161b109be77cd4f30ce0662bfe8cc32/cli/registry/client/client.go#L24-L25 (but conversely, doesn't make you think about details like Accept headers )