Multiarch Docker

[yarons]

Docker supports multiarchitecture build nowadays, which means you can upload several docker architecture variants for the same tag to Docker hub.

This can all be done using a well configured Docker's buildx.

I have a working example on my GH repo which uploads the multiarch artifact to ghcr.io instead of Docker hub:
https://github.com/yarons/pritunl-zero-docker/actions/runs/1736367592/workflow
https://github.com/yarons/pritunl-zero-docker/blob/main/.github/workflows/docker-image.yml

These are the actual artifacts (ready for pull in both architectures):
https://github.com/yarons?tab=packages&repo_name=pritunl-zero-docker

[szaimen]

Hi, thanks for your idea! Unfortunately we cannot build multiarch images as we rely on the docker repository api which itself is not able to handle multiarch container images. See #42 (comment)
So this is unfortunately a won't fix for the time being.

[yarons]

@szaimen So how do you explain this? (Or am I missing something?)
https://hub.docker.com/_/mongo?tab=tags

[szaimen]

Yes, docker hub is able to display multiarch images. I don't know if they implemented a custom, not documented api or how they are doing it. Feel free to investigate how you can retreive the checksums for multiarch images. See #42 (comment) for some initial steps

[szaimen]

If you find a way, we may consider building multiarch images.

[yarons]

@szaimen The pipelines in my GitHub are modified for ghcr but I can adapt these to Dockerhub (The original script was for Dockerhub but I prefer ghcr so I modified it), where are your pipelines running?

If we were to modify my pipelines all you need to do is to replace the name of the image to dockerhub instead of ghcr and login to Dockerhub (The username and password should be stored as Docker secrets), then run docker push.

[szaimen]

No, that is not what I am talking about. Did you read the comments that I've linked to?

[szaimen]

The problem is that we are not able to retreive the checksums of multiarch images via the docker repo api

[szaimen]

I know that it is possible to create multiarch container images without much trouble

[yarons]

I see but why do you care so much for the hashes? Docker is agile enough to pull the right image to the right hosting arch.

[szaimen]

Because we need that in order to be able to judge if the containers are up-to-date or not ;)

[szaimen]

See e.g.

all-in-one/php/src/Docker/DockerActionManager.php

Lines 73 to 85 in 4e85276

	public function GetContainerUpdateState(Container $container) : IContainerState
	{
	$tag = $this->GetCurrentChannel();
	$runningDigest = $this->GetRepoDigestOfContainer($container->GetIdentifier());
	$remoteDigest = $this->dockerHubManager->GetLatestDigestOfTag($container->GetContainerName(), $tag);
	if ($runningDigest === $remoteDigest) {
	return new VersionEqualState();
	} else {
	return new VersionDifferentState();
	}
	}

[yarons]

I saw something about tag architecture in here, could that possibly help?
https://www.arthurkoziel.com/dockerhub-registry-api/
(Get Image Manifest section).

[szaimen]

Thanks but that is kind of actually what we are doing already: see

all-in-one/php/src/Docker/DockerHubManager.php

Lines 37 to 45 in 4e85276

	'https://registry-1.docker.io/v2/'.$name.'/manifests/' . $tag,
	[
	'headers' => [
	'Accept' => 'application/vnd.docker.distribution.manifest.v2+json',
	'Authorization' => 'Bearer ' . $authToken,
	],
	]
	);
	$responseHeaders = $manifestRequest->getHeader('docker-content-digest');

So as I said: until you show me a way to get all needed hashes of multiarch images, this will not be implemented :)

[yarons]

I see, it looks like although there's architecture field in his responsed I couldn't find that officially.

I'll get back to you if I'll find something, thanks.

[szaimen]

Thank you, too! :)

[yarons]

I think I found something, run the following command on your machine and look at the output:
docker manifest inspect --verbose library/ubuntu:latest

There must be a way, I'll try to see if I can get some documentation for it but I can definitely see multiarch hashes.

On the docker docs there's a warning saying that this is experimental.

[szaimen]

Hi, I just found out that it is possible to get all digest by querying this endpoint:

https://registry.hub.docker.com/v2/repositories/user/repository/tags

example

curl -L -s 'https://registry.hub.docker.com/v2/repositories/library/centos/tags?page_size=1024'

{
  "creator": 1156886,
  "id": 105191642,
  "images": [
    {
      "architecture": "amd64",
      "features": "",
      "variant": null,
      "digest": "sha256:fd84102fc72960dd1b8da0ee3b4c13e3b0c1d2a085de118bc4c97821cd986e02",
      "os": "linux",
      "os_features": "",
      "os_version": null,
      "size": 74866818,
      "status": "active",
      "last_pulled": "2023-01-05T20:52:45.259516Z",
      "last_pushed": null
    },
    {
      "architecture": "ppc64le",
      "features": "",
      "variant": null,
      "digest": "sha256:77eebe55df5e0521188b7fa224feddb1877b8c40901173e0f87aa3a97ecef4d2",
      "os": "linux",
      "os_features": "",
      "os_version": null,
      "size": 84620966,
      "status": "active",
      "last_pulled": "2023-01-05T21:09:32.02647Z",
      "last_pushed": null
    },

However creating multi-arch images will also mean that testing gets much slower since I would alway need to wait until the multi-arch image is built which takes up to 45min compared to around 5min since Github Actions does not allow for hardware-accellaration. So I'd rather wait for that to be available before taking another look at this.

[yarons]

Basically the "multiarch" approach is merely adding a single tag to 2 different hashes, if there's a way to modify tag after its being created then I can possibly add the first hash first and the second hash when it's ready, this way it'll take longer but at least one of the builds will be available.

[Zoey2936]

Or using this part: „The hard way with docker manifest“ from here: https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

[Zoey2936]

docker manifest create nextcloud/all-in-one:latest -a nextcloud/all-in-one:latest -a nextcloud/all-in-one:latest-arm64
docker manifest push nextcloud/all-in-one:latest
could work

[szaimen]

What about keeping the build like it is and only pushing the multiarch to beta and latest by using docker manifest create? This could probably be added here https://github.com/nextcloud-releases/all-in-one/blob/9b649b6e1705abafb95b1828bb873632ec318e7d/.github/workflows/promote-to-beta.yml#L50 and and here https://github.com/nextcloud-releases/all-in-one/blob/9b649b6e1705abafb95b1828bb873632ec318e7d/.github/workflows/promote-to-latest.yml#L83

[szaimen]

TODO:

• merge and test Build multiarch nextcloud-releases/all-in-one#31
• get multiarch repodigest #1887
• adjust the documentation (only when latest multiarch is provided)

[Zoey2936]

Even if I think this should work, I don't understand how watchtower works with digests of multiarch, since they use another API (so it works with repos outside docker hub)

[szaimen]

yeah, not sure either

[szaimen]

I think they have multiple techniques. In worst case they fall back to manual pulling a new image iirc...

[Zoey2936]

https://docs.docker.com/registry/spec/manifest-v2-2/ was the problem maybe that the mastercontainer didnt accepted the "application/vnd.docker.distribution.manifest.list.v2+json" manifest?

[szaimen]

I think it is fine for now to only accept docker hub images. We can later on still improve it.

[szaimen]

So basically everything was merged and built correclty but the digest did not match:
nextcloud-releases/all-in-one#31 (comment)

So not sure how to proceed from here. I reverted the multiarch building and now the update check works correctly again. So it might depend on how exactly the manifests get created but not sure how to retrieve the correct digest. This needs more testing.

[szaimen]

This is now released with v4.3.1 Beta. Testing and feedback is welcome! See https://github.com/nextcloud/all-in-one#how-to-switch-the-channel