Repository navigation
executor: a starting builder no longer holds prepare, and no docker command fails because of it - #52
Merged
Merged
Conversation
buildx create --driver remote saves the instance and then probes the
endpoint under its --timeout, 20 s by default. While the project's builder
is still booting nothing answers on its port, and the remote driver reports
it inactive without an error, so the create waited the full 20 s and then
succeeded anyway. Every job that found its builder booting started its
script 20 s late.
For a booting builder the script now runs the create with --timeout 2s.
If that create fails, the unbounded create runs once more: buildx before
v0.32.0 has no --timeout on create, and a builder that turns ready during
the probe can make the bounded create roll back. A create that still fails
is reported as before ("could not attach the builder") and the job builds
without the cache. Ready builders get the same create as before. The docker
wrapper, which waits for a starting builder on docker build, is unchanged.
The tests run the generated script under bash with fake docker, ip and ssh
binaries: bounded create succeeds, fails then the fallback succeeds (old
buildx, rolled-back create), both fail, and a ready builder's create
succeeding and failing.
The docker wrapper waited for a starting builder only for docker build
and docker buildx. With BUILDX_BUILDER set to a builder that does not
answer yet, docker compose build does not build locally: it fails after
about 40 s ("waiting for connection ... context deadline exceeded"). The
20 s create in prepare used to give the builder that much head start;
with the bounded create a compose build right after prepare fails the job.
The wrapper now also waits for docker compose build and for compose up,
run or create with --build. It skips compose's options that take a value
to find the command, so "compose --profile build up" does not wait. It
uses the same wait as docker build: up to BuilderWait, then it unsets
BUILDX_BUILDER and compose builds locally. A compose up that builds a
missing image without --build is not seen and does not wait; docs/jobs.md
says so, and that buildx older than v0.32.0 still waits about 20 s.
Tests:
- The wrapper the builder script installs runs under bash for docker
build, buildx and compose commands: the builder answers, never answers
(local build), was already found up or down, and commands that must not
wait.
- The fake docker refuses a second create of an instance, like buildx,
and a case covers an instance that already exists.
- The tests fail if the script uses a guest path they do not redirect.
The docker wrapper only protected the commands on its build list. Any other command kept BUILDX_BUILDER on a builder that did not answer yet, and a build it ran anyway failed after about 40 s instead of building locally. The common case is docker compose up building a missing image on a fresh VM; docker --debug build, docker --context x compose build, docker image build and docker builder build were missed as well. Until the builder has answered once, a command the wrapper does not see as a build now runs with BUILDX_BUILDER unset, without probing or waiting: such a build is local and keeps the job alive. Once a build has found the builder up, every command uses it. Recognised builds wait as before. The build list also grows. The wrapper skips docker's global options before the command, so docker --debug build and docker --context x compose build are seen. image build and builder build are aliases of buildx build, so they are builds too. Compose's hidden --workdir is skipped as an option that takes a value. Tests: - New wrapper cases while the builder starts: compose up, push and run run locally at once, and image build, builder build and builds after docker options wait. - After the builder answered, compose up uses it. - The fake docker logs "local" only when BUILDX_BUILDER is unset. - The attach test checks that the script is hermetic before Run starts.
Only the wrapper's probe wrote /run/firerunner-builder-ok, and only builds the wrapper recognises run the probe. With a ready builder, a docker compose up that builds a missing image without --build therefore ran locally and cold, where before this branch it used the warm cache. When the daemon reports the builder ready, the builder script now writes /run/firerunner-builder-ready. A command the wrapper does not see as a build keeps BUILDX_BUILDER if the builder is ready or has answered. It runs without the builder while the builder starts, and always once it was given up. Recognised builds still probe the builder first, so a ready builder that went away since prepare still falls back to a local build. Tests: - Ready-state wrapper cases: compose up and run use the builder without a probe, compose up after the builder was given up is local, and a build probes, then is remote or local. - The unit test checks that only a ready builder's script writes the marker.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a job's project builder was still booting, prepare spent 20 s in
docker buildx create: buildx saves the instance, then probes the endpoint for--timeout(default 20 s), and the remote driver reports an unreachable endpoint as inactive without an error, so the probe always ran to the limit.docker buildx create --timeout 2s; any failure falls back to one unbounded create (buildx older than v0.32.0 behaves as before). A ready builder keeps today's command and its script marks the builder ready.docker build,buildx,image build,builder build,compose buildandcompose up/run/create --build, also after docker's global options. If the builder never answers, the build runs locally, asdocker buildalready did.docker compose buildagainst a builder that did not answer yet failed the job after about 40 s.Measured in a Firecracker microVM (buildx v0.37.1, Compose v5.5.1): bounded create against an unreachable builder returns in 2.2–2.6 s (was 20.1 s); compose against an inactive builder fails after 40 s, and builds locally in 6 s once
BUILDX_BUILDERis unset.Tests cover the create paths (bounded, fallback, both failing, existing instance, ready) and the wrapper for booting and ready builders, with hermetic fakes; an end-to-end run against real docker and a real mTLS BuildKit passed for both states.