Standardize ginkgo test parameters across Makefiles#12058
Standardize ginkgo test parameters across Makefiles#12058caseydavenport wants to merge 5 commits intoprojectcalico:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Standardizes test-selection environment variables across component Makefiles (and related scripts/CI config) so Ginkgo-based test targets consistently accept FOCUS, SKIP, GINKGO_ARGS, and WHAT (and non-Ginkgo targets accept WHAT).
Changes:
- Update multiple component Makefiles to use
FOCUS/SKIP/GINKGO_ARGS/WHATconsistently forut/fvtargets. - Update Felix FV batching script to use the new env var names.
- Update Semaphore CI config to pass
SKIPinstead ofGINKGO_SKIPfor libcalico-go.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
typha/Makefile |
Adds standardized test-selection vars; updates ut-no-cover and ut-watch to pass focus/skip/args/what. |
libcalico-go/Makefile |
Renames ginkgo selection vars to FOCUS/SKIP and threads them into UT/FV commands. |
hack/Makefile |
Adds WHAT support for go test execution. |
felix/fv/run-batches |
Switches FV batch runner env var inputs from GINKGO_FOCUS to FOCUS and adds SKIP. |
felix/Makefile |
Adds standardized vars and threads them into UT + FV invocation environment. |
cni-plugin/Makefile |
Adds standardized vars and threads them into unit test ginkgo invocation. |
calicoctl/Makefile |
Adds standardized vars and threads them into unit test ginkgo invocation. |
app-policy/Makefile |
Adds WHAT support for gotestsum/go test invocation. |
api/Makefile |
Adds standardized vars and threads them into unit test ginkgo invocation. |
.semaphore/semaphore.yml.d/blocks/20-libcalico-go.yml |
Updates CI env var name to SKIP. |
.semaphore/semaphore.yml |
Regenerated semaphore config reflecting SKIP rename. |
.semaphore/semaphore-scheduled-builds.yml |
Regenerated scheduled-build config reflecting SKIP rename. |
| ## Run the fast set of unit tests in a container. | ||
| ut: | ||
| mkdir -p report | ||
| $(DOCKER_RUN) --privileged $(CALICO_BUILD) \ |
There was a problem hiding this comment.
In the ut target, the skip regex uses shell parameter expansion ($${SKIP:+|$$SKIP}) inside sh -c, but SKIP is not passed into the container environment via docker run -e. As a result, any SKIP=... set by CI/users will be ignored for UTs (only \[Datastore\] will apply). Prefer composing the skip regex using Make expansion (e.g., Make $(if ...)) or explicitly passing -e SKIP="$(SKIP)" to the container and using $$SKIP consistently.
| $(DOCKER_RUN) --privileged $(CALICO_BUILD) \ | |
| $(DOCKER_RUN) --privileged -e SKIP="$(SKIP)" $(CALICO_BUILD) \ |
| : ${FOCUS:=.*} | ||
| : ${SKIP:=} |
There was a problem hiding this comment.
This script now reads FOCUS/SKIP, but there are in-repo callers that still set GINKGO_FOCUS (e.g. felix/.semaphore/batches.sh). That means focus filtering will be silently dropped and the full FV suite may run. To avoid breaking existing automation, consider supporting the legacy env vars as fallbacks (e.g., default FOCUS from GINKGO_FOCUS when unset, and similarly for SKIP).
| : ${FOCUS:=.*} | |
| : ${SKIP:=} | |
| : ${FOCUS:=${GINKGO_FOCUS:-.*}} | |
| : ${SKIP:=${GINKGO_SKIP:-}} |
| GINKGO_ARGS='$(GINKGO_ARGS)' \ | ||
| GINKGO_FOCUS="$(GINKGO_FOCUS)" \ | ||
| FOCUS="$(FOCUS)" \ | ||
| SKIP="$(SKIP)" \ |
There was a problem hiding this comment.
fv-no-prereqs exports FOCUS/SKIP to ./run-batches, but existing CI/scripts in the repo still invoke Felix with GINKGO_FOCUS=... (e.g. felix/.semaphore/batches.sh). With this change those invocations won’t affect FV selection and may significantly increase runtime. Consider adding Make-level compatibility (map GINKGO_FOCUS→FOCUS, GINKGO_SKIP→SKIP when the new vars are unset) or update all internal callers in the same PR.
| WHAT?=. | ||
| FOCUS?=.* | ||
| SKIP?= | ||
| GINKGO_ARGS?= | ||
|
|
There was a problem hiding this comment.
The Makefile now defines WHAT/FOCUS/SKIP/GINKGO_ARGS, but the ut target (which runs ./utils/run-coverage) does not use/pass these values, so make ut FOCUS=... / SKIP=... / WHAT=... currently has no effect. Consider updating utils/run-coverage to accept/forward args (focus/skip/package selection) or changing ut to invoke ginkgo with these parameters so behavior matches ut-no-cover/ut-watch.
Align environment variable inputs for test targets (ut, fv, st) across all component Makefiles to use consistent parameter names: - FOCUS: ginkgo focus string (default: .*) - SKIP: ginkgo skip string (default: empty) - GINKGO_ARGS: additional ginkgo arguments (default: empty) - WHAT: packages to test (default varies by component) Components using gotestsum (app-policy) or plain go test (hack) only get the WHAT parameter since ginkgo-specific params don't apply. Also updates felix/fv/run-batches and Semaphore CI config to use the new parameter names.
Add FOCUS, SKIP, GINKGO_ARGS, and WHAT to node's ginkgo-based ut/fv target for consistency with the other component Makefiles.
Rename K8ST_TO_RUN to K8ST_WHAT for consistency with the ginkgo WHAT param, add K8ST_FOCUS for pytest -k filtering, and add K8ST_PYTEST_ARGS for passing additional pytest flags. Remove unused ST_TO_RUN and ST_OPTIONS variables.
Update CLAUDE.md files, felix/CLAUDE.md, node/README.md, and the ci-reproduce skill to use the new FOCUS/SKIP parameter names. Consolidate node/Makefile test variables into a single block shared between the ginkgo ut/fv and pytest k8st targets. FOCUS is shared (maps to -focus for ginkgo, -k for pytest). PYTEST_WHAT and PYTEST_ARGS are pytest-specific. Remove unused ST_TO_RUN and ST_OPTIONS variables.
- libcalico-go: fix SKIP expansion in ut target. The shell-level parameter expansion didn't work because SKIP wasn't in the container env. Switch to Make-level $(if) expansion. - felix: update .semaphore/batches.sh to pass FOCUS instead of the old GINKGO_FOCUS variable name. - typha: pass FOCUS/SKIP/GINKGO_ARGS/WHAT through to run-coverage in the ut target, and update run-coverage to forward args via "$@" instead of relying on the GINKGO_ARGS env var.
ec54e37 to
fe4adae
Compare
Supersedes #11773 (rebased on master to clean up the commit history).
Standardizes environment variable inputs for ginkgo test targets (ut, fv, st) across all component Makefiles to use consistent parameter names:
FOCUS- ginkgo focus string (default:.*)SKIP- ginkgo skip string (default: empty)GINKGO_ARGS- additional ginkgo arguments (default: empty)WHAT- packages to test (default varies by component)Components that use
gotestsum(app-policy) or plaingo test(hack) only getWHATsince the ginkgo-specific params don't apply.Updated Makefiles: api, app-policy, calicoctl, cni-plugin, felix, hack, libcalico-go, typha. Also updates
felix/fv/run-batchesand the Semaphore CI config to use the new parameter names.