Parallelize kind-build-images for faster kind-up#12069
Draft
caseydavenport wants to merge 3 commits intoprojectcalico:masterfrom
Draft
Parallelize kind-build-images for faster kind-up#12069caseydavenport wants to merge 3 commits intoprojectcalico:masterfrom
caseydavenport wants to merge 3 commits intoprojectcalico:masterfrom
Conversation
marvin-tigera
added
release-note-required
caseydavenport
force-pushed
the
caseydavenport/parallel-kind-build-images
branch
3 times, most recently
from
0a2f1f8 to
ce113c0
Compare
Build all component images concurrently by invoking kind-build-images with make -j$(nproc). The stamp file targets are independent of each other, so they can safely run in parallel. The operator build already depends on all image markers, so it correctly serializes after them. Apply the same treatment to kind-reload.
Replace the docker-save/kind-load-image-archive approach with a local Docker registry (localhost:5001). Images are pushed to the registry after building, and kind nodes pull from it directly. This is faster than the tarball approach because docker push transfers only changed layers, and avoids the overhead of creating and loading a multi-GB archive. It also more closely mirrors how images are pulled in a real cluster. Changes: - Add kind-img-registry target to start a local registry container - Configure containerd on kind nodes to mirror localhost:5001 to the registry container - Push images to registry in kind-build-images and build-operator.sh - Update values.yaml and calicoctl.yaml to use localhost:5001 registry - Switch operator pull policy from PullNever to PullAlways - Add push_images.sh, used by deploy_resources.sh
Two changes: 1. build-operator.sh: Reuse existing operator clone across runs instead of deleting and re-cloning every time. On subsequent runs, just fetch+reset to latest. This preserves the Go build cache, downloaded helm binary, and Istio/Gateway API charts. 2. node/Makefile: Exclude build artifacts (*.ll, *.o, *.d, libbpf/) from the GPL source tarball and its find-based dependency list. The BPF compilation creates transient .ll files in felix/bpf-gpl/ that cause tar to fail when running in parallel (-j) because files appear and disappear mid-archive.
caseydavenport
force-pushed
the
caseydavenport/parallel-kind-build-images
branch
from
ce113c0 to
640753b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
kind-upandkind-reloadby invokingkind-build-imageswithmake -j$(nproc)instead of sequentially.lib.Makefileare independent of each other, so they cansafely run in parallel. The operator build depends on all image markers, so it
correctly serializes after them.
Test plan
make kind-upfrom a clean state (no stamp files) and verify all images buildand the cluster comes up healthy.
make kind-reloadafter a source change and verify only the changed image rebuilds.