kamal build push fails Docker Hub auth with macOS alias

[mblayman]

tl;dr - kamal build push failed to auth with Docker Hub when using the macOS alias, but it succeeded when I ran kamal via a local gem install on the host OS.

Notably, the local gem install run includes a couple of [auth] steps that were not visible in the previous run that used the macOS alias.

---

Context:

• Mac mini M1, 2020
• macOS Sonoma 14.6.1
• Docker Desktop 4.34.0 (latest as of writing this issue)
• Auth creds in "credsStore": "desktop"

I am using kamal from the macOS alias of

alias kamal='docker run -it --rm -v "${PWD}:/workdir" -v "/run/host-services/ssh-auth.sock:/run/host-services/ssh-auth.sock" -e SSH_AUTH_SOCK="/run/host-services/ssh-auth.sock" -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/basecamp/kamal:latest'

that is listed in the install docs. After preparing an app, my demo repo at https://github.com/mblayman/kamal-demo was failing when I tried to send the image to Docker Hub using kamal build push. I was authenticated already and proved that I was authenticated by pushing an alpine image to https://hub.docker.com/repository/docker/mblayman/kamal-demo/general to convince myself that my auth was valid.

When running kamal build push, I had a log like (trimming some of the less relevant bits):

❯ kamal build push
  INFO [078570c5] Running docker --version && docker buildx version on localhost
  INFO [078570c5] Finished in 0.174 seconds with exit status 0 (successful).
  INFO Cloning repo into build directory `/tmp/kamal-clones/kamal-demo-2f65914456263/workdir/`...
  INFO [239e06e4] Running /usr/bin/env git -C /tmp/kamal-clones/kamal-demo-2f65914456263 clone /workdir --recurse-submodules on localhost
  INFO [239e06e4] Finished in 0.029 seconds with exit status 0 (successful).
  INFO [7cb74728] Running /usr/bin/env git -C /tmp/kamal-clones/kamal-demo-2f65914456263/workdir/ status --porcelain on localhost
  INFO [7cb74728] Finished in 0.002 seconds with exit status 0 (successful).
  INFO [ed1051ea] Running /usr/bin/env git -C /tmp/kamal-clones/kamal-demo-2f65914456263/workdir/ rev-parse HEAD on localhost
  INFO [ed1051ea] Finished in 0.001 seconds with exit status 0 (successful).
  INFO [e3d316a0] Running docker buildx inspect kamal-kamal-demo-multiarch > /dev/null on localhost
  WARN Missing compatible builder, so creating a new one first
  INFO [17699192] Running docker buildx create --use --name kamal-kamal-demo-multiarch on localhost
  INFO [17699192] Finished in 0.110 seconds with exit status 0 (successful).
  INFO [290e6195] Running docker buildx build --push --platform linux/amd64,linux/arm64 --builder kamal-kamal-demo-multiarch -t mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981 -t mblayman/kamal-demo:latest --label service="kamal-demo" --file Dockerfile . on localhost
 DEBUG [290e6195] Command: docker buildx build --push --platform linux/amd64,linux/arm64 --builder kamal-kamal-demo-multiarch -t mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981 -t mblayman/kamal-demo:latest --label service="kamal-demo" --file Dockerfile .
 DEBUG [290e6195] 	#0 building with "kamal-kamal-demo-multiarch" instance using docker-container driver
<snip> 
 DEBUG [290e6195] 	#16 exporting to image
 DEBUG [290e6195] 	#16 exporting layers 0.0s done
 DEBUG [290e6195] 	#16 exporting manifest sha256:f40f357163181e70c234e75649405b12e940dc4a38701d986c262ac62972b4ed
 DEBUG [290e6195] 	#16 exporting manifest sha256:f40f357163181e70c234e75649405b12e940dc4a38701d986c262ac62972b4ed done
 DEBUG [290e6195] 	#16 exporting config sha256:ff2ff768e4dbfc542935c3e3f47d96ca6ee222c2a4e32cd5d281bf91c0a048d6 done
 DEBUG [290e6195] 	#16 exporting attestation manifest sha256:98765dec991583bd37c7fea41a5ae2bfebae8da365b0af11e904ac4a6b197554 done
 DEBUG [290e6195] 	#16 exporting manifest sha256:e6192b8dad16021b7068139744f47c87f572470ce06d12afa38be07fc38d1239 done
 DEBUG [290e6195] 	#16 exporting config sha256:1bb4610975c62f4b8361374b4c21674f17f9a10806f299b7570e5fc738767f64 done
 DEBUG [290e6195] 	#16 exporting attestation manifest sha256:41fc891dc6fe39443909a675678bc3c0c7f91ef20282f6fe0e55ed5d923ad289 done
 DEBUG [290e6195] 	#16 exporting manifest list sha256:c975ef3d679c406e4557815d66227fdf9f6ec1ceb10f529d2e5f0c53c88352e7 done
 DEBUG [290e6195] 	#16 pushing layers
 DEBUG [290e6195] 	#16 pushing layers 0.9s done
 DEBUG [290e6195] 	#16 ERROR: failed to push mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
 DEBUG [290e6195] 	------
 DEBUG [290e6195] 	 > exporting to image:
 DEBUG [290e6195] 	------
 DEBUG [290e6195] 	ERROR: failed to solve: failed to push mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  ERROR (SSHKit::Command::Failed): docker exit status: 256
docker stdout: Nothing written
<snip>

All the details are viewable via the stream where I did this live at https://www.youtube.com/live/mJPSSUVL4qw

This evening (24 hours later), I restarted the machine (since I had just installed Docker in the same session as kamal), re-ran kamal build push, and confirmed that I still had the same behavior.

Then I proceeded to install kamal via the brew installed version of Ruby. I repeated kamal build push and it succeeded with no modifications to any configuration files or files in the demo repo!

Here is the relevant log:

❯ kamal build push
  INFO [45bc142b] Running docker --version && docker buildx version as matt@localhost
  INFO [45bc142b] Finished in 0.178 seconds with exit status 0 (successful).
  INFO Cloning repo into build directory `/var/folders/gx/5g0y_8_x73x78lm7g3bknqym0000gn/T/kamal-clones/kamal-demo-ace65239dacb9/kamal-demo/`...
  INFO [3aae22b3] Running /usr/bin/env git -C /var/folders/gx/5g0y_8_x73x78lm7g3bknqym0000gn/T/kamal-clones/kamal-demo-ace65239dacb9 clone /Users/matt/projects/kamal-demo --recurse-submodules as matt@localhost
  INFO [3aae22b3] Finished in 0.545 seconds with exit status 0 (successful).
  INFO [7623a79b] Running /usr/bin/env git -C /var/folders/gx/5g0y_8_x73x78lm7g3bknqym0000gn/T/kamal-clones/kamal-demo-ace65239dacb9/kamal-demo/ status --porcelain as matt@localhost
  INFO [7623a79b] Finished in 0.009 seconds with exit status 0 (successful).
  INFO [718a37c7] Running /usr/bin/env git -C /var/folders/gx/5g0y_8_x73x78lm7g3bknqym0000gn/T/kamal-clones/kamal-demo-ace65239dacb9/kamal-demo/ rev-parse HEAD as matt@localhost
  INFO [718a37c7] Finished in 0.006 seconds with exit status 0 (successful).
  INFO [75487003] Running docker buildx inspect kamal-kamal-demo-multiarch > /dev/null as matt@localhost
  WARN Missing compatible builder, so creating a new one first
  INFO [3711649c] Running docker buildx create --use --name kamal-kamal-demo-multiarch as matt@localhost
  INFO [3711649c] Finished in 0.123 seconds with exit status 0 (successful).
  INFO [ffbe207c] Running docker buildx build --push --platform linux/amd64,linux/arm64 --builder kamal-kamal-demo-multiarch -t mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981 -t mblayman/kamal-demo:latest --label service="kamal-demo" --file Dockerfile . as matt@localhost
 DEBUG [ffbe207c] Command: docker buildx build --push --platform linux/amd64,linux/arm64 --builder kamal-kamal-demo-multiarch -t mblayman/kamal-demo:7a5e89f209f7f16a6d2d9ec239931b1817c33981 -t mblayman/kamal-demo:latest --label service="kamal-demo" --file Dockerfile .
 DEBUG [ffbe207c] 	#0 building with "kamal-kamal-demo-multiarch" instance using docker-container driver
<snip> 
 DEBUG [ffbe207c] 	#3 [auth] library/python:pull token for registry-1.docker.io
 DEBUG [ffbe207c] 	#3 DONE 0.0s
<snip>
 DEBUG [ffbe207c] 	#17 exporting to image
 DEBUG [ffbe207c] 	#17 exporting layers 0.0s done
 DEBUG [ffbe207c] 	#17 exporting manifest sha256:96ddba416c55ecf9e386dacb3b1f865b214b79b5470d2242ce953ebe986a0035 done
 DEBUG [ffbe207c] 	#17 exporting config sha256:6ce2ccf6e3237775ddbc91507380ac6322d2ee245a0ef5923e70d618892a1313 done
 DEBUG [ffbe207c] 	#17 ...
 DEBUG [ffbe207c]
 DEBUG [ffbe207c] 	#18 [auth] mblayman/kamal-demo:pull,push token for registry-1.docker.io
 DEBUG [ffbe207c] 	#18 DONE 0.0s
 DEBUG [ffbe207c]
<snip>
 DEBUG [ffbe207c] 	#17 DONE 117.8s
 DEBUG [ffbe207c]
 DEBUG [ffbe207c] 	View build details: docker-desktop://dashboard/build/kamal-kamal-demo-multiarch/kamal-kamal-demo-multiarch0/sjoa0z8g329c48mr48t0bgo1p
  INFO [ffbe207c] Finished in 127.938 seconds with exit status 0 (successful).

Notably, this run includes a couple of [auth] steps that were not visible in the previous run that used the macOS alias.

My best working theory is that the Dockerized version of kamal did not have access to the auth information that was set on the host OS, but I don't know the inner workings of kamal. I hope this report helpful.

[djmb]

kamal build push when run by itself doesn't try to login to the docker registry so whether it succeeds or not depends on whether you are logged in already. The docker version of the command won't be.

kamal deploy it will log you in to docker hub so that does work from within Docker.

[mblayman]

Thank you, @djmb! This makes sense to me. Is this by design?

I'm new to kamal so I'm trying to determine some of the philosophy around working with the tool. I like to ease into a new tool by working incrementally. Because of that preference, I attempted to do what I consider the first half of the flow (i.e., create an app/kamal setup that uploads an image artifact to a registry).

My next goal was to move into using that artifact on a cloud host.

It sounds like the only way to have that approach currently is to use the non-Dockerized version of kamal (which is fine by me). Is the intent to have the Dockerized version work at parity with the non-Dockerized version? Or will there always be some limitations around the Dockerized version that should be documented?