Tag signatures seem to be broken

[tomaszzielinski]

It seems that something is wrong with signing base Node images' tags on Docker Hub:

$ DOCKER_CONTENT_TRUST=1 docker pull node
Using default tag: latest
Pull (1 of 1): node:latest@sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b
sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b: Pulling from library/node
Digest: sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b
Status: Image is up to date for node@sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b
Tagging node@sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b as node:latest

$ docker images --digests node | grep latest
node                latest              sha256:f0b151a0f4226e67e40afddbc9e0a37e697f8eb32d5cd8b1a55d5b238f04581b   9ba05fbb174a        5 months ago        900MB

$ docker pull node
Using default tag: latest
latest: Pulling from library/node
Digest: sha256:cd932f9ff15650a908bf5982c7c0c5aa032d378edcc5cf179a3f7fc8bc8683ef
Status: Downloaded newer image for node:latest

$ docker images --digests node | grep latest
node                latest              sha256:6e64f63a663a368cc81b28ed3c3e29e6b3784c04f0128be5aaa659157ed4d231   7c412a558705        12 days ago         907MB
node                latest              sha256:cd932f9ff15650a908bf5982c7c0c5aa032d378edcc5cf179a3f7fc8bc8683ef   7c412a558705        12 days ago         907MB

As you can see the signed node:latest tag is outdated while the unsigned one is recent (and matches node:12.6.0).
(By the way, there seem to be two digests for the same image, hopefully that's fine?)

Also, it appears that the recent images have no signed counterparts:

$ DOCKER_CONTENT_TRUST=1 docker pull node:12.6.0
No valid trust data for 12.6.0

But images pushed roughly 5 months ago are still signed:

$ DOCKER_CONTENT_TRUST=1 docker pull node:10.15
Pull (1 of 1): node:10.15@sha256:7050e0dffc069c9f2e8dedcd255d7e57d87bebf33d6a5d97bd4905fb2333db8c
sha256:7050e0dffc069c9f2e8dedcd255d7e57d87bebf33d6a5d97bd4905fb2333db8c: Pulling from library/node
Digest: sha256:7050e0dffc069c9f2e8dedcd255d7e57d87bebf33d6a5d97bd4905fb2333db8c
Status: Image is up to date for node@sha256:7050e0dffc069c9f2e8dedcd255d7e57d87bebf33d6a5d97bd4905fb2333db8c
Tagging node@sha256:7050e0dffc069c9f2e8dedcd255d7e57d87bebf33d6a5d97bd4905fb2333db8c as node:10.15

[LaurentGoderre]

@tianon any idea?

[tianon]

The process described in docker-library/official-images#1516 is still accurate (in other words, we don't have any involvement or insight into the signing process); I'll poke internally at Docker and see if someone can take a look at what's up. 😞

[tomaszzielinski]

Just for the record, it appears that similar problems happened before:

docker-library/official-images#5874
getsentry/docker-sentry#173

[dpn]

@tianon Any luck with the poking?

[tianon]

docker-library/official-images#5874 (comment) was the (successful) result of my last round of poking.

[dyihoon90]

Still facing the issue, so we decided to pull via the image digest, gotten from here

@tianon am i right to say this should be as safe as using DCT, but just that we have to be more involved in our node upgrade process compared to using docker tags?

bash-3.2$ export DOCKER_CONTENT_TRUST=1 

bash-3.2$ docker pull node:12.9.0-alpine
No valid trust data for 12.9.0-alpine

bash-3.2$ docker pull node@sha256:b1a32805dbe3894722db93b7f1a3489a208ddf4e9b9341d1e8d3e366fcc0998d
sha256:b1a32805dbe3894722db93b7f1a3489a208ddf4e9b9341d1e8d3e366fcc0998d: Pulling from library/node
Digest: sha256:b1a32805dbe3894722db93b7f1a3489a208ddf4e9b9341d1e8d3e366fcc0998d
Status: Image is up to date for node@sha256:b1a32805dbe3894722db93b7f1a3489a208ddf4e9b9341d1e8d3e366fcc0998d
docker.io/library/node@sha256:b1a32805dbe3894722db93b7f1a3489a208ddf4e9b9341d1e8d3e366fcc0998d

[tianon]

I don't believe notary is consulted at all when you pull by digest, but I'd generally recommend pulling by digest anyhow if you want to make sure you're getting the exact bits you expect to be (especially over time).

[tomaszzielinski]

@dyihoon90 My understanding is that DCT bind tags to digests, so you can be sure that pulling a tag gives you a legitimate image. And if you pull directly by the digest then the end result is the same, but the process is less convenient.

(Of course it also depends on the proper implementation of digest verification in the Docker client--if that part is broken then I guess there can be no guarantees at all.)

[tesharp]

Anyone looking into this at the moment? Since we rely on trusted images this is quite a big deal...

[muktihari]

I encounter the same issue, no content trust for images other than the latest. Any idea?
$ echo $DOCKER_CONTENT_TRUST && docker pull node:12.18.3-slim 1 No valid trust data for 12.18.3-slim
$ echo $DOCKER_CONTENT_TRUST && docker pull node:14.11.0-buster-slim 1 No valid trust data for 14.11.0-buster-slim

[tianon]

Unfortunately this is still docker-library/official-images#6838, and there's not anything we can do about it without Docker's help. 😞

[muktihari]

Thanks @tianon for reaching out, yeah I read the link yesterday. I tried to file an issue in docker forum, I hope it will get attention there. https://forums.docker.com/t/dct-enabled-on-node-official-images-return-no-valid-trust-data/99197