Implement Docker caching on top of BuildKit

[Kobzol]

We cache all Docker containers built on rust-lang/rust CI on S3, to make rebuilds pretty much instant. This happens here. In August 2023, GitHub updated their Docker version, which then started defaulting to the BuildKit Docker backend. This backend doesn't support our way of exporting the Docker layer SHAs (moby/buildkit#1235), therefore after this update, our caching stopped working, which immediately made our CI workflows run for much longer.

I "fixed" this in rust-lang/rust#114763, where I just opted in to using the old Docker build backend. However, it is marked as legacy, and it will eventually stop working. When that happens, we should have some other solution for using a cache with BuildKit.

2023-10-09T06:02:41.6917009Z Attempting with retry: docker build --rm -t rust-ci -f /home/runner/work/rust/rust/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile /home/runner/work/rust/rust/src/ci/docker
2023-10-09T06:02:41.6995446Z DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
2023-10-09T06:02:41.6995904Z             BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
2023-10-09T06:02:41.6997019Z             environment-variable.

There is supposed to be a S3 cache backend for Docker with BuildKit (https://docs.docker.com/build/cache/backends/s3/), but it's marked as experimental currently, and it needs some custom Docker driver, which I'm not sure if it works on GHA. There is also GHA cache available for Docker, but it's not applicable to us, since it has a 10 GiB size limit (AFAIK).

Note: the way our caching currently works, we have to export all the intermediate Docker layers. It's not enough to just export the last layer. Because Docker performs the build layer by layer, and if it won't find an existing image for the first layer, it will just rebuild everything from scratch (in other words, it does not know the SHA of the final layer until it builds the penultimate layer).

[Kobzol]

Trying to fix this in rust-lang/rust#114762. So far I'm having problems with the BuildKit S3 cache backend not having the correct permissions to access an S3 bucket.

[Mark-Simulacrum]

https://aws.amazon.com/blogs/containers/announcing-remote-cache-support-in-amazon-ecr-for-buildkit-clients/

This might also be interesting to explore in the future.

[Kobzol]

I implemented caching using the ghcr.io registry here. It stores the cached Docker images under ghcr.io/rust-lang-ci/rust-ci. It seems to work for try/auto builds, PR builds don't have the necessary permissions on their GITHUB_TOKEN, which is probably a good thing, and it has also worked like this previously.