Skip to content

Standardize ginkgo test parameters across Makefiles#12058

Open
caseydavenport wants to merge 5 commits intoprojectcalico:masterfrom
caseydavenport:cd4/standardize-ginkgo-params
Open

Standardize ginkgo test parameters across Makefiles#12058
caseydavenport wants to merge 5 commits intoprojectcalico:masterfrom
caseydavenport:cd4/standardize-ginkgo-params

Conversation

@caseydavenport
Copy link
Member

Supersedes #11773 (rebased on master to clean up the commit history).

Standardizes environment variable inputs for ginkgo test targets (ut, fv, st) across all component Makefiles to use consistent parameter names:

  • FOCUS - ginkgo focus string (default: .*)
  • SKIP - ginkgo skip string (default: empty)
  • GINKGO_ARGS - additional ginkgo arguments (default: empty)
  • WHAT - packages to test (default varies by component)

Components that use gotestsum (app-policy) or plain go test (hack) only get WHAT since the ginkgo-specific params don't apply.

Updated Makefiles: api, app-policy, calicoctl, cni-plugin, felix, hack, libcalico-go, typha. Also updates felix/fv/run-batches and the Semaphore CI config to use the new parameter names.

# Examples
make -C felix fv FOCUS="BPF"
make -C felix ut FOCUS="Wireguard"
make -C libcalico-go ut SKIP="SlowTest"
make -C felix ut GINKGO_ARGS="-ginkgo.v"
make -C api ut WHAT="./pkg/client"

@caseydavenport caseydavenport requested review from a team as code owners March 10, 2026 22:16
@caseydavenport caseydavenport added docs-not-required Docs not required for this change release-note-not-required Change has no user-facing impact labels Mar 10, 2026
Copilot AI review requested due to automatic review settings March 10, 2026 22:16
@marvin-tigera marvin-tigera added this to the Calico v3.32.0 milestone Mar 10, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Standardizes test-selection environment variables across component Makefiles (and related scripts/CI config) so Ginkgo-based test targets consistently accept FOCUS, SKIP, GINKGO_ARGS, and WHAT (and non-Ginkgo targets accept WHAT).

Changes:

  • Update multiple component Makefiles to use FOCUS/SKIP/GINKGO_ARGS/WHAT consistently for ut/fv targets.
  • Update Felix FV batching script to use the new env var names.
  • Update Semaphore CI config to pass SKIP instead of GINKGO_SKIP for libcalico-go.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
typha/Makefile Adds standardized test-selection vars; updates ut-no-cover and ut-watch to pass focus/skip/args/what.
libcalico-go/Makefile Renames ginkgo selection vars to FOCUS/SKIP and threads them into UT/FV commands.
hack/Makefile Adds WHAT support for go test execution.
felix/fv/run-batches Switches FV batch runner env var inputs from GINKGO_FOCUS to FOCUS and adds SKIP.
felix/Makefile Adds standardized vars and threads them into UT + FV invocation environment.
cni-plugin/Makefile Adds standardized vars and threads them into unit test ginkgo invocation.
calicoctl/Makefile Adds standardized vars and threads them into unit test ginkgo invocation.
app-policy/Makefile Adds WHAT support for gotestsum/go test invocation.
api/Makefile Adds standardized vars and threads them into unit test ginkgo invocation.
.semaphore/semaphore.yml.d/blocks/20-libcalico-go.yml Updates CI env var name to SKIP.
.semaphore/semaphore.yml Regenerated semaphore config reflecting SKIP rename.
.semaphore/semaphore-scheduled-builds.yml Regenerated scheduled-build config reflecting SKIP rename.

## Run the fast set of unit tests in a container.
ut:
mkdir -p report
$(DOCKER_RUN) --privileged $(CALICO_BUILD) \
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the ut target, the skip regex uses shell parameter expansion ($${SKIP:+|$$SKIP}) inside sh -c, but SKIP is not passed into the container environment via docker run -e. As a result, any SKIP=... set by CI/users will be ignored for UTs (only \[Datastore\] will apply). Prefer composing the skip regex using Make expansion (e.g., Make $(if ...)) or explicitly passing -e SKIP="$(SKIP)" to the container and using $$SKIP consistently.

Suggested change
$(DOCKER_RUN) --privileged $(CALICO_BUILD) \
$(DOCKER_RUN) --privileged -e SKIP="$(SKIP)" $(CALICO_BUILD) \

Copilot uses AI. Check for mistakes.
Comment on lines +23 to +24
: ${FOCUS:=.*}
: ${SKIP:=}
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script now reads FOCUS/SKIP, but there are in-repo callers that still set GINKGO_FOCUS (e.g. felix/.semaphore/batches.sh). That means focus filtering will be silently dropped and the full FV suite may run. To avoid breaking existing automation, consider supporting the legacy env vars as fallbacks (e.g., default FOCUS from GINKGO_FOCUS when unset, and similarly for SKIP).

Suggested change
: ${FOCUS:=.*}
: ${SKIP:=}
: ${FOCUS:=${GINKGO_FOCUS:-.*}}
: ${SKIP:=${GINKGO_SKIP:-}}

Copilot uses AI. Check for mistakes.
Comment on lines 380 to +382
GINKGO_ARGS='$(GINKGO_ARGS)' \
GINKGO_FOCUS="$(GINKGO_FOCUS)" \
FOCUS="$(FOCUS)" \
SKIP="$(SKIP)" \
Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fv-no-prereqs exports FOCUS/SKIP to ./run-batches, but existing CI/scripts in the repo still invoke Felix with GINKGO_FOCUS=... (e.g. felix/.semaphore/batches.sh). With this change those invocations won’t affect FV selection and may significantly increase runtime. Consider adding Make-level compatibility (map GINKGO_FOCUSFOCUS, GINKGO_SKIPSKIP when the new vars are unset) or update all internal callers in the same PR.

Copilot uses AI. Check for mistakes.
Comment on lines +138 to +142
WHAT?=.
FOCUS?=.*
SKIP?=
GINKGO_ARGS?=

Copy link

Copilot AI Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Makefile now defines WHAT/FOCUS/SKIP/GINKGO_ARGS, but the ut target (which runs ./utils/run-coverage) does not use/pass these values, so make ut FOCUS=... / SKIP=... / WHAT=... currently has no effect. Consider updating utils/run-coverage to accept/forward args (focus/skip/package selection) or changing ut to invoke ginkgo with these parameters so behavior matches ut-no-cover/ut-watch.

Copilot uses AI. Check for mistakes.
Align environment variable inputs for test targets (ut, fv, st) across all
component Makefiles to use consistent parameter names:

- FOCUS: ginkgo focus string (default: .*)
- SKIP: ginkgo skip string (default: empty)
- GINKGO_ARGS: additional ginkgo arguments (default: empty)
- WHAT: packages to test (default varies by component)

Components using gotestsum (app-policy) or plain go test (hack) only get
the WHAT parameter since ginkgo-specific params don't apply.

Also updates felix/fv/run-batches and Semaphore CI config to use the new
parameter names.
Add FOCUS, SKIP, GINKGO_ARGS, and WHAT to node's ginkgo-based ut/fv
target for consistency with the other component Makefiles.
Rename K8ST_TO_RUN to K8ST_WHAT for consistency with the ginkgo WHAT
param, add K8ST_FOCUS for pytest -k filtering, and add K8ST_PYTEST_ARGS
for passing additional pytest flags.

Remove unused ST_TO_RUN and ST_OPTIONS variables.
Update CLAUDE.md files, felix/CLAUDE.md, node/README.md, and the
ci-reproduce skill to use the new FOCUS/SKIP parameter names.

Consolidate node/Makefile test variables into a single block shared
between the ginkgo ut/fv and pytest k8st targets. FOCUS is shared
(maps to -focus for ginkgo, -k for pytest). PYTEST_WHAT and
PYTEST_ARGS are pytest-specific. Remove unused ST_TO_RUN and
ST_OPTIONS variables.
- libcalico-go: fix SKIP expansion in ut target. The shell-level
  parameter expansion didn't work because SKIP wasn't in the container
  env. Switch to Make-level $(if) expansion.
- felix: update .semaphore/batches.sh to pass FOCUS instead of the old
  GINKGO_FOCUS variable name.
- typha: pass FOCUS/SKIP/GINKGO_ARGS/WHAT through to run-coverage in
  the ut target, and update run-coverage to forward args via "$@"
  instead of relying on the GINKGO_ARGS env var.
@caseydavenport caseydavenport force-pushed the cd4/standardize-ginkgo-params branch from ec54e37 to fe4adae Compare March 12, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required Docs not required for this change release-note-not-required Change has no user-facing impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants