Repository navigation
Cache misses using a remote build cache when sharing docker targets #3878
Description
Activity
I hit this too and narrowed it down a bit. It doesn't seem to be specific to a shared stage or to registry caches. It reproduces with
type=localcaches whenever two targets share a local mount: the same main context directory, or the same named context. Each target exports to its own cache. On every rebuild with a fresh builder, one of the two targets re-runs a step it should have taken from cache, and which target misses varies between runs. If the second target points at the same directory through a symlink, so the mounts no longer have the same path, both targets hit on every rebuild.That matches
detectSharedMountsinbuild/build.go, which puts requests whose local mounts have the same(name, dir)into one shared session. In your repro,alphaandbetasharecontexts = { api = "./api" }, which is the same condition. This is a guess from reading the code; I haven't traced where the cache match gets lost.Self-contained script (bash,
docker-containerdriver, no registry). There is no shared stage; the onlyRUNuses a per-targetARG:repro.sh
#!/usr/bin/env bash # Two bake targets, each with its own local cache, rebuilt on a fresh builder. # Prints which targets re-ran a step that should have come from cache. set -euo pipefail cd "$(mktemp -d)" mkdir -p ctx ctx-a ctx-b shared echo a > ctx/f; echo a > ctx-a/f; echo a > ctx-b/f; echo s > shared/f ln -s ctx ctx-link cat > Dockerfile <<'EOF' FROM alpine:3.20 ARG T COPY f /f RUN echo "EXECUTED $T" && date > /out EOF cat > Dockerfile.named <<'EOF' FROM alpine:3.20 ARG T COPY --from=shared f /f RUN echo "EXECUTED $T" && date > /out EOF # $1 = scenario name, $2 = context of a, $3 = context of b, $4 = dockerfile, $5 = named contexts bakefile() { cat <<EOF group "default" { targets = ["a", "b"] } target "_common" { dockerfile = "$PWD/$4" output = ["type=cacheonly"] contexts = { $5 } } target "a" { inherits = ["_common"] context = "$2" args = { T = "a" } cache-from = ["type=local,src=$PWD/cache-$1-a"] cache-to = ["type=local,dest=$PWD/cache-$1-a,mode=max"] } target "b" { inherits = ["_common"] context = "$3" args = { T = "b" } cache-from = ["type=local,src=$PWD/cache-$1-b"] cache-to = ["type=local,dest=$PWD/cache-$1-b,mode=max"] } EOF } scenario() { bakefile "$@" > "$1.hcl" for run in 1 2 3 4; do docker buildx create --name "repro-$1-$run" --driver docker-container >/dev/null docker buildx bake --builder "repro-$1-$run" --progress=plain \ --allow=fs.read="$PWD" --allow=fs.write="$PWD" -f "$1.hcl" > "$1-$run.log" 2>&1 docker buildx rm "repro-$1-$run" >/dev/null 2>&1 reran=$(grep -oE '^#[0-9]+ [0-9.]+ EXECUTED [ab]' "$1-$run.log" | awk '{print $4}' | sort | tr '\n' ' ' || true) echo "$1 run $run (fresh builder): re-ran: ${reran:-none}" done } scenario shared-context ./ctx ./ctx Dockerfile "" scenario symlinked-context ./ctx ./ctx-link Dockerfile "" scenario shared-named-context ./ctx-a ./ctx-b Dockerfile.named 'shared = "./shared"'
Output (run 1 is the cold build that fills each cache):
shared-context run 1 (fresh builder): re-ran: a b shared-context run 2 (fresh builder): re-ran: b shared-context run 3 (fresh builder): re-ran: b shared-context run 4 (fresh builder): re-ran: b symlinked-context run 1 (fresh builder): re-ran: a b symlinked-context run 2 (fresh builder): re-ran: none symlinked-context run 3 (fresh builder): re-ran: none symlinked-context run 4 (fresh builder): re-ran: none shared-named-context run 1 (fresh builder): re-ran: a b shared-named-context run 2 (fresh builder): re-ran: a shared-named-context run 3 (fresh builder): re-ran: a shared-named-context run 4 (fresh builder): re-ran: bVersions: buildx
v0.30.1-desktop.1, BuildKitv0.33.1(moby/buildkit:buildx-stable-1), Docker Desktop 29.1.2, macOS arm64.detectSharedMountsis unchanged on master (42b3f95). We see the same thing in GitHub Actions withtype=ghacaches and two compose services built from the same context directory.Workaround until there's a fix: give each target its own context path (a symlink to the shared directory is enough).
Contributing guidelines
I've found a bug and checked that ...
Description
We are affected by an interesting remote cache problem. We get cache hits for each individual target, but not when multiple targets are build at the same time.
In the linked repo:
docker buildx bakedocker builder prune -fdocker buildx bake --progress=plain alpha betaThis will usually result in something like:
So a cache hit for one build but not for the other.
However executing
2.
docker buildx bake3.
docker builder prune -f4.
docker buildx bake --progress=plain alpha5.
docker builder prune -f6.
docker buildx bake --progress=plain betaWill result in a full cache hit and never print the statement
Expected behaviour
docker buildx bake alpha betashould have the same cache hits as
Actual behaviour
1 Target gets a cache hit, the other targets dont.
Buildx version
github.com/docker/buildx v0.33.0-desktop.1 7f91f038ac14cbf5c4b2a6b76470860814424da1
Docker info
Builders list
Configuration
I made a reproducer here:
https://github.com/tg-freigmbh/repro-bake/tree/main
Build logs
Additional info
This seems to be very much tight to the docker buildtarget "build".
Something along the lines that due to the ARG SVC it could result in 2 different images, but in fact it doesnt.
For our productive problem we were able to solve this by inlining the target "build" into "runtime"-> Since docker lazy resolves ARG and shares everything anyway this seems currently as a good workaround without downsides.