Dependency is not part of the same cache chain

[jakeyheath]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

We run docker buildx in our CICD pipelines using Github Actions. We've used the same Github Workflows for several years now, with the default caching behavior set to inline caching and storing our images in Amazon ECR. Recently, we upgraded our docker buildx version and started see errors like:

Error: buildx failed with: ERROR: failed to build: failed to solve: dependency sha256:e2aa7bbca7e0480ed9d31d7951c3300a24651faff1d5bc9718905c46ea394f40 is not part of the same cache chain

I couldn't find any documentation about this error. It looks like this error message was introduced recently, but I don't know what it means. For some context, we have at least two --cache-from statements. The first cache-from points to a tag from the current branch and the second cache-from is the tag from the main branch. Ideally on the first commit, they will have a cache miss from their branch and pick up the cache from the main branch. On subsequent commits, they will pick up the cache from their branch.

Since this issue, we have changed from inline caching to registry caching and haven't seen the error return. I appreciate any help understanding the error message and correcting whats wrong with our caching setup.

Reproduction

I am having a hard time providing instructions for reproducing the issue. Here are the flags that were used in the failing CI job:

--build-arg IMAGE_TAG=<GIT_SHORT_SHA> \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from type=registry,ref=<ECR>:branch-<FEATURE_BRANCH> \
--cache-from type=registry,ref=<ECR>:branch-<MAIN> \
--cache-to type=inline,mode=max \
--file Dockerfile \
--iidfile /home/runner/_work/_temp/docker-actions-toolkit-ohXfcK/build-iidfile-a8dbc73217.txt \
--label org.opencontainers.image.created=2025-09-23T23:52:49.106Z \
--label org.opencontainers.image.description=<DESC> \
--label org.opencontainers.image.licenses=MIT \
--label org.opencontainers.image.revision=<GIT_REV> \
--label org.opencontainers.image.source=<GIT_REPO> \
--label org.opencontainers.image.title=<GIT_REPO_NAME> \
--label org.opencontainers.image.url=<GIT_REPO> \
--label org.opencontainers.image.version=<GIT_BRANCH> \
--platform linux/amd64 \
--tag <ECR>:branch-<FEATURE_BRANCH> \
--tag <ECR>:<GIT_SHORT_SHA> \
--tag <ECR>:<GIT_LONG_SHA> \
--load \
--metadata-file /home/runner/_work/_temp/docker-actions-toolkit-ohXfcK/build-metadata-dac4a221c4.json \
--push .

The issue seemed to happen more frequently on multi-stage builds where only the last cache layer is stored. Users reported that after re-running the job several times, sometimes the error would go away and they were able to complete the job.

Version information

/usr/bin/docker version
  Client: Docker Engine - Community
   Version:           28.4.0
   API version:       1.51
   Go version:        go1.24.7
   Git commit:        d8eb465
   Built:             Wed Sep  3 20:57:05 2025
   OS/Arch:           linux/amd64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          28.4.0
    API version:      1.51 (minimum version 1.24)
    Go version:       go1.24.7
    Git commit:       249d679
    Built:            Wed Sep  3 20:58:50 2025
    OS/Arch:          linux/amd64
    Experimental:     false
   containerd:
    Version:          v1.7.28
    GitCommit:        b98a3aace656320842a23f4a392a33f46af97866
   runc:
    Version:          1.3.0
    GitCommit:        v1.3.0-0-g4ca628d
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    28.4.0
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.26.1
      Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.39.4
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 1
    Running: 1
    Paused: 0
    Stopped: 0
   Images: 3
   Server Version: 28.4.0
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: true
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   CDI spec directories:
    /etc/cdi
    /var/run/cdi
   Swarm: inactive
   Runtimes: runc io.containerd.runc.v2
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: b98a3aace656320842a23f4a392a33f46af97866
   runc version: v1.3.0-0-g4ca628d
   init version: de40ad0
   Security Options:
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.1.148-173.267.amzn2023.x86_64
   Operating System: Alpine Linux v3.22 (containerized)
   OSType: linux
   Architecture: x86_64
   CPUs: 48
   Total Memory: 371.7GiB
   Name: amd64-privileged-6jgxk-runner-62h75
   ID: 02ced34e-dbee-4110-ab45-a455145b13ad
   Docker Root Dir: /home/runner/_work/docker
   Debug Mode: false
   Experimental: false
   Insecure Registries:
    ::1/128
    127.0.0.0/8
   Live Restore Enabled: false
   Product License: Community Engine

  /usr/bin/docker buildx version
  github.com/docker/buildx v0.26.1 1a8287f22cf5a38339a4c1bf432b803c5f8b2aae

[tonistiigi]

I'm not sure if the version info is correct, as this should be a new error in the v0.25 release.

It's internal consitency check that is failing, like one cache export(or provenance generation) is leaking to another. Would be helpful to understand the conditions for reproduction.

[jakeyheath]

Looks like our buildkit version is using the "master" tag, so maybe we just pulled in the tip of the repo instead of a stable release version. We should probably update that on our end. The ECR repo is just a pull-through cache from dockerhub.

[jakeyheath]

Would be helpful to understand the conditions for reproduction.

Our team needed a quick fix to unblock builds, so its hard to reproduce, but I will try today to make a commit based on the old version of the CI. Maybe I can get it to trigger again.

[jakeyheath]

Ok here is some updated builder information from one of the failed builds:

Booting builder
  /usr/bin/docker buildx inspect --bootstrap --builder builder-211995d6-f581-4f44-9359-c44421b8730a
  #1 [internal] booting buildkit
  #1 pulling image 533267185808.dkr.ecr.us-west-2.amazonaws.com/docker.io/central/moby/buildkit:master
  #1 pulling image 533267185808.dkr.ecr.us-west-2.amazonaws.com/docker.io/central/moby/buildkit:master 2.5s done
  #1 creating container buildx_buildkit_builder-211995d6-f581-4f44-9359-c44421b8730a0
  #1 creating container buildx_buildkit_builder-211995d6-f581-4f44-9359-c44421b8730a0 1.4s done
  #1 DONE 3.9s
  Name:          builder-211995d6-f581-4f44-9359-c44421b8730a
  Driver:        docker-container
  Last Activity: 2025-09-24 20:34:02 +0000 UTC
  
  Nodes:
  Name:                  builder-211995d6-f581-4f44-9359-c44421b8730a0
  Endpoint:              unix:///var/run/docker.sock
  Driver Options:        image="533267185808.dkr.ecr.us-west-2.amazonaws.com/docker.io/central/moby/buildkit:master"
  Status:                running
  BuildKit daemon flags: --allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host
  BuildKit version:      c8fad61
  Platforms:             linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386
  Labels:
   org.mobyproject.buildkit.worker.executor:         oci
   org.mobyproject.buildkit.worker.hostname:         1148a50c7c89
   org.mobyproject.buildkit.worker.network:          host
   org.mobyproject.buildkit.worker.oci.process-mode: sandbox
   org.mobyproject.buildkit.worker.selinux.enabled:  false
   org.mobyproject.buildkit.worker.snapshotter:      overlayfs
  GC Policy rule#0:
   All:            false
   Filters:        type==source.local,type==exec.cachemount,type==source.git.checkout
   Keep Duration:  48h0m0s
   Max Used Space: 488.3MiB
  GC Policy rule#1:
   All:            false
   Keep Duration:  1440h0m0s
   Reserved Space: 9.313GiB
            "keepDuration": "48h0m0s",
            "maxUsedSpace": "488.3MiB"
          },
          {
            "all": false,
            "keepDuration": "1440h0m0s",
            "reservedSpace": "9.313GiB",
            "maxUsedSpace": "93.13GiB",
            "minFreeSpace": "46.57GiB"
          },
          {
            "all": false,
            "reservedSpace": "9.313GiB",
            "maxUsedSpace": "93.13GiB",
            "minFreeSpace": "46.57GiB"
          },
          {
            "all": true,
            "reservedSpace": "9.313GiB",
            "maxUsedSpace": "93.13GiB",
            "minFreeSpace": "46.57GiB"
          }
        ]
      }
    ],
    "name": "builder-211995d6-f581-4f44-9359-c44421b8730a",
    "driver": "docker-container",
    "lastActivity": "2025-09-24T20:34:02.000Z"
  }
BuildKit version
  builder-211995d6-f581-4f44-9359-c44421b8730a0: c8fad61

[jakeyheath]

So I have tried to recreate it, but can't seem to pin it down. I swapped the master tag for the tag in the above comment and tried to cherry pick our inline cache settings from before I updated them to use registry caching. But every time I run the CI again, it seems to work this time around. I'm sorry I don't have a good reproduction for you. If you want, I can package up some of the logs for the old failing logs for the Github Action if you think that would be helpful. If not, feel free to disregard this issue

[tonistiigi]

If you have debug logs or stacktrace from the error then please send.

[tonistiigi]

It seems that the conditions for this were the usage of multiple exporters (both --load and --push in your example). #6263 should fix it.