Build failures on arm64, error "ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:..." from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:...: not found

[monkeyhandz]

All buildpack builds are failing on Apple arm64. The error message is always the same (the SHA does change):

ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:6528df63e9cafd1747f04fc57cad7b3910dce6be0e6cf49e8174fb2aeda69ef6" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:2fc1acc6fd5a97e82e2b780700a2c07e813e84fac1fe5857b42f475e6dc526d0: not found

We have tried numerous cleans and pruning of images etc with no success.

Below is a logs of trying to build the java-maven example from samples (https://github.com/buildpacks/samples).

Environment details

pack report

$ pack report
Pack:
  Version:  0.37.0+git-bf1bd85.build-6370
  OS/Arch:  darwin/arm64

Default Lifecycle Version:  0.20.3

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12, 0.13

Config:
(no config file found at /Users/smit005/.pack/config.toml)

docker info

docker info
Client:
 Version:    28.0.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.21.1-desktop.2
    Path:     /Users/smit005/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.33.1-desktop.1
    Path:     /Users/smit005/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.38
    Path:     /Users/smit005/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.5
    Path:     /Users/smit005/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/smit005/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     /Users/smit005/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/smit005/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/smit005/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/smit005/.docker/cli-plugins/docker-sbom

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 32
 Server Version: 28.0.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.14-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 7.654GiB
 Name: docker-desktop
 ID: 8339c278-9b66-4609-b342-44c18608b794
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/smit005/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Build log:

$ pack build myapp --builder cnbs/sample-builder:noble

Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded.
For more information, see https://github.com/buildpacks/pack/issues/2272.
noble: Pulling from cnbs/sample-builder
5a7813e071bf: Pull complete
1763d0c6967f: Pull complete
15c3891f682c: Pull complete
9257a508d8b4: Pull complete
2dfaa2cde8a8: Pull complete
738cf76368f9: Pull complete
6c6c63e26408: Pull complete
fd112c64004f: Pull complete
4f4fb700ef54: Pull complete
3882072c5ab9: Pull complete
36e0c996ff9f: Pull complete
49025e280f6c: Pull complete
8ca48f841f8c: Pull complete
295156d243be: Pull complete
a2e47f292e2e: Pull complete
de305c158623: Pull complete
6bd59d79929c: Pull complete
1e96ded835bc: Pull complete
Digest: sha256:9cad4157f28edc9df97e2396868d18a1f75df48fd437da5c255c9aa1ea8e485f
Status: Downloaded newer image for cnbs/sample-builder:noble
noble: Pulling from cnbs/sample-base-run
2ba9a37ab3bf: Pull complete
Digest: sha256:c04b351655724608568dc59c99cfcc1825501d2302ac9a7d7e5e77b2c041b6ea
Status: Downloaded newer image for cnbs/sample-base-run:noble
0.20.3: Pulling from buildpacksio/lifecycle
5c5654c6d510: Pull complete
a62778643d56: Pull complete
5664b15f108b: Pull complete
da7816fa955e: Pull complete
9aee425378d2: Pull complete
0bab15eea81d: Pull complete
c6b97f964990: Pull complete
4aa0ea1413d3: Pull complete
7c12895b777b: Pull complete
33e068de2649: Pull complete
bfb59b82a9b6: Pull complete
8ffb3c3cf71a: Pull complete
Digest: sha256:6528df63e9cafd1747f04fc57cad7b3910dce6be0e6cf49e8174fb2aeda69ef6
Status: Downloaded newer image for buildpacksio/lifecycle:0.20.3
ERROR: failed to build: failed to fetch base layers: saving image with ID "sha256:6528df63e9cafd1747f04fc57cad7b3910dce6be0e6cf49e8174fb2aeda69ef6" from the docker daemon: Error response from daemon: unable to create manifests file: NotFound: content digest sha256:2fc1acc6fd5a97e82e2b780700a2c07e813e84fac1fe5857b42f475e6dc526d0: not found
$

On Page: Understand build failures

[AidanDelaney]

Thank you for this. I can't readily replicate it, but I'll confer with others to do so.

[monkeyhandz]

@AidanDelaney I see this is on the docs project, should I raise it against the pack project instead?

[natalieparellano]

This is probably the same as buildpacks/pack#2270. We were not able to get to the bottom of this, but perhaps it is time for further investigation

[monkeyhandz]

Is there any update on this issue?

[GeorgeJose7]

+1

[caicaishmily]

Same with your issue.

[1n4001]

I'm getting the same issue. RHEL8 w/ podman. Using DOCKER_HOST={path/to/}podman.sock doing a normal Spring bootBuildImage. My issue was resolved by doing dnf install podman-docker.

[AidanDelaney]

Intersting @monkeyhandz does installing podman-docker help in your case?

[AT92]

+1

[schapp1]

+1

[AT92]

For me it helped to disable in docker desktop the containerd option

[mrohlof-protofy]

This is a workaround, though, not a real solution. This issue does not happen with all buildpacks, too. For me, builder-noble-java-tiny works fine without the fix, while run-jammy-full does not.

[jmesny]

Same problem with builder-jammy-base... 😕

[zageyiff]

I'm having this exact same issue on Mac OS X, 15.6.1, arch: aarch64, m2 pro, using colima

❯ colima status -p default
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] docker socket: unix:///Users/user/.colima/default/docker.sock
INFO[0000] containerd socket: unix:///Users/user/.colima/default/containerd.sock

(somehow I ended up with two sockets, one for docker one for containerd, even though my runtime is docker)

Couldn't find a way to fix or workaround it using colima, so I uninstalled colima and installed Docker for Desktop (not ideal for me), and disable containerd as @AT92 mentioned, and it's working for me, at least now I'm not blocked for this.

Edit:
I dig a little deeper as I'd like to continue using colima. Starting with colima v0.8.3, docker info shows the storage driver type is set to (with default config)

Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1

abiosoft/colima#958 (comment)
This causes this warning Warning: Exporting to docker daemon (building without --publish) and daemon uses containerd storage; performance may be significantly degraded., and whenever I have this warning, I get the issue from this thread.

To fix it, I deleted my colima profile (or create a new profile), and re-created it by editing the config before creating/starting the colima VM colima start --edit ..., and disabled containerd-snapshotter by adding this in the config.yaml

docker:
  features:
    buildkit: true
    containerd-snapshotter: false

now doing a docker info shows me this instead and I can use pack with colima in apple sillicon chip

 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false

I have no use case for containerd-snapshotter for now so I'm fine disabling it