Repository navigation
docker driver loses BuildKit job for large bake target sets #3810
Description
Activity
That looks legit, but right now it overlaps pretty heavily with #3486. The interesting failure here is not really: "reading docker-bake.hcl took a long time", because that progress line is just the local bake definition loading step. The useful signal is the later:
failed to receive status: rpc error: code = NotFound desc = no such job ...Can you please reduce this to a raw
buildx bakerepro and attach the exact outputs below? That will tell us whether this is a Bake matrix scaling bug, a BuildKit job lifecycle bug under load, or just a Desktop-specific failure mode.Please try this minimal repro first.
target "default" { name = "${a}${b}-${c}" tags = ["${a}${b}:latest${c}"] dockerfile-inline = "FROM scratch" matrix = { a = [1,2,3,4,5,6,7,8,9,10] b = [1,2,3,4,5,6,7,8,9,10] c = [1,2,3] } }
Then run these commands and paste the full output.
docker buildx version docker version docker info docker buildx inspect --bootstrap docker buildx bake --print docker buildx -D bake --progress=plain
It would also help a lot if you bisect the matrix size and report the smallest failing case. For example, please try
10x10x1,10x10x2,10x10x3, and then smaller combinations until you find the first size that fails reliably.If this only fails with many targets, please say whether it still fails when using a non-Desktop builder, for example a
docker-containerbuilder instead of the defaultdockerdriver.If you want to rule out the default
dockerdriver, please try the same repro with a separatedocker-containerbuilder.Example:
docker buildx create --name repro-container --driver docker-container --use docker buildx inspect --bootstrap docker buildx bake --print docker buildx -D bake --progress=plain
If you want to switch back afterwards:
docker buildx use default
If you want to remove the test builder afterwards:
docker buildx rm repro-container
Please include the output of:
docker buildx ls docker buildx inspect --bootstrap
for both the default builder and the
docker-containerbuilder, and say clearly whether the failure reproduces in both cases or only with one of them.Right now my guess is that this is the same class of issue as #3486, but we need the raw
buildx bakerepro and debug output to prove that.Files collected using the requested repro using default/desktop-linux driver
bootstrap.log
dversion.log
info.log
plain.log
print.log
version.logFiles collected using the requested repro using docker-container driver
Build takes about 80s to start, but failure is not reproducible (works as expected, but slow)I tried switching back back to default to reavaluate:
Thorsten Arbeitsplat@DESKTOP-EQRL39B MINGW64 /g/development/bake-bug (master) $ docker buildx use default ERROR: run `docker context use default` to switch to default contextIs that expected?
After switching back using context use default: rpc failure is still reproducibleSo I converted the matrix to a an explicit file (no matrix feature):
Results are about the same: its slow but succeeds using the docker-container-driver.
Its slow and fails using the desktop driver.
So there are probably 2 separate tasks:- bake should be faster when there are many targets (in hcl files?)
- the desktop driver should not fail if bake is a bit slower
From my experimentation:
100 targets work (but also slow),
200-300 targets do not work.But might be down to individual cpu performance
I tried switching back back to default to reavaluate:
Thorsten Arbeitsplat@DESKTOP-EQRL39B MINGW64 /g/development/bake-bug (master) $ docker buildx use default ERROR: run `docker context use default` to switch to default contextIs that expected? After switching back using context use default: rpc failure is still reproducible
Ah for this in your case as you're on WSL it would be:
docker buildx use desktop-linuxFiles collected using the requested repro
This is much better, thanks. The latest repro narrows the issue a lot.
The key observations now are that the failure reproduces with raw
buildx bake, it still reproduces after expanding the matrix into an explicit bake file, and it only fails with the defaultdocker/ Desktop path whiledocker-containeris slow but succeeds.That means this no longer looks matrix-specific. The matrix feature is just one way to generate a large number of targets. The correctness bug here appears to be that the
dockerdriver loses the job under a larger Bake target set and returns:failed to receive status: rpc error: code = NotFound desc = no such job ...The rough threshold you found, around
100works and200-300fails, is also useful.So I would reframe this as a driver-specific correctness issue under larger Bake target counts, with a separate performance concern on top.
This sounds very close to the same class of failure as #3486, but the new repro here is cleaner because it isolates the default
dockerdriver versusdocker-container.So I think the next step is to focus on why the
dockerdriver path is dropping the BuildKit job for larger Bake target sets. I think thedockerdriver exposes an engine-side BuildKit bug/limit.Build takes about 80s to start, but failure is not reproducible (works as expected, but slow)
For
docker-container, tune BuildKit instead of using the embedded driver. BuildKit supportsmax-parallelismin daemon config. A typical config for the container driver is:[worker.oci] max-parallelism = 16
Then create the builder with that config:
docker buildx create \ --name large-bake \ --driver docker-container \ --buildkitd-config /path/to/buildkitd.toml \ --use --bootstrap
If you want to you can create a default config file in buildx config (
~/.docker/buildx/buildkitd.default.toml) so it's looked up when creating builders: https://docs.docker.com/reference/cli/docker/buildx/create/#buildkitd-configThat is a mitigation for throughput and resource pressure. It's not the root fix for the
dockerdriver bug though.- changed the title
[-]very poor performance using matrix builds will cause rpc failures[/-][+]docker driver loses BuildKit job for large bake target sets[/+]on Apr 23, 2026 Ok more oddities...
I tried the same thing on a linux machine.using the desktop driver: its fast and fails fast getting the same rpc error ( so its not about a job timing out as I initially suspected)
using the container driver: its faster than on windows, but seems t spend an unreasonable amount of time handling Dockerfile and dockerignores... but maybe thats related to the micro being a bit stupid...In eihter case max-parallelism doesnt seem to affect perfomance noticbly, wheter using 16 or 1
Reacted by CrazyMaxThere might be some bottleneck in here that would be nice to debug but ultimately I think in these extreme cases only setting some upper limit helps. Every build makes at least 3 grpc requests in parallel, and if they can't synchronize with each other because of some overload, some timeout needs to follow (current session job timeouts that this seems to hit are quite short).
Easiest probably would be to add some
max-buildslimit in buildkit config, default unset or some high number. That could queue the requests without any processing if too many active builds come in at the same time.We do have a bench that triggers 128 builds in parallel in https://moby.github.io/buildkit-bench/ . These are small builds, but in that it does seem to figure them all out in ~7sec. If all builds can actually start, then the
max-parallelismwould already take effect on scheduling the individual steps.I can add that at 120 target it sometimes works, sometimes doesnt...
This change may help out with this: moby/buildkit#6741
It won't necessarily solve it completely, but it'll remove an area where buildkit could be the reason why the 3 second timeout wasn't reached.
Contributing guidelines
I've found a bug and checked that ...
Description
bakes matrix build has very poor performance. If you create > 100 targets using a matrix bake will spend > 1 min in
=> => reading docker-bake.hcl 235B / 235BThis will ultimatively cause the build to fail with an error message like
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5aI havent eveluated if the matrix feature is the culprit here or if this always happens if there is a nontrivial amount of targets.
Expected behaviour
Better performance, or the build should at least not fail
Actual behaviour
After about 3 minutes the build fails:
ERROR: target 58-1: failed to receive status: rpc error: code = NotFound desc = no such job xtcwi97x2fguo5z68ffeg6v5aBuildx version
github.com/docker/buildx v0.33.0-desktop.1 7f91f038ac14cbf5c4b2a6b76470860814424da1
Docker info
Builders list
Configuration
Build logs
Additional info
No response