Add YAML-based test selection config for e2e tests#12350
Open
caseydavenport wants to merge 2 commits intoprojectcalico:masterfrom
Open
Add YAML-based test selection config for e2e tests#12350caseydavenport wants to merge 2 commits intoprojectcalico:masterfrom
caseydavenport wants to merge 2 commits intoprojectcalico:masterfrom
Conversation
Add a testconfig package that loads YAML config files defining which e2e tests to include/exclude using Ginkgo v2 labels and test name patterns. Config files support single-parent inheritance via 'extends', with child configs appending to parent include/exclude lists. The e2e test binary accepts --calico.test-config=<path> to load a config file and apply the resulting label-filter and skip patterns to the ginkgo suite config before running tests. Convert the kind-based e2e-run-test Makefile target to use this instead of the old E2E_FOCUS/E2E_SKIP make variables.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a YAML-driven mechanism for selecting e2e tests (include/exclude) and wires it into the local kind-based make e2e-run-test flow, replacing hard-to-maintain ginkgo focus/skip regex strings.
Changes:
- Added
e2e/pkg/testconfigto load YAML configs (withextendsinheritance) and convert them into Ginkgo v2 label-filter + skip patterns, with unit tests. - Added initial YAML configs under
e2e/config/(base/kind/gcp-bpf). - Updated the e2e test binary to accept
--calico.test-config, and updatedMakefileto pass the config to the e2e test run.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
Makefile |
Replaces E2E_FOCUS/E2E_SKIP with E2E_TEST_CONFIG and passes --calico.test-config into the e2e test binary. |
e2e/pkg/testconfig/types.go |
Defines YAML schema/types and validation helpers for includes/excludes. |
e2e/pkg/testconfig/loader.go |
Implements YAML loading + extends resolution + structural validation. |
e2e/pkg/testconfig/convert.go |
Converts resolved config into Ginkgo label-filter + skip regex string(s). |
e2e/pkg/testconfig/testconfig_test.go |
Unit tests covering loading/extends/validation and flag conversion. |
e2e/pkg/config/config.go |
Registers --calico.test-config flag and exposes TestConfigPath(). |
e2e/config/base.yaml |
Base shared include/exclude label rules for CI-style runs. |
e2e/config/kind.yaml |
Kind-focused selection config used by make e2e-run-test. |
e2e/config/gcp-bpf.yaml |
GCP+BPF selection config extending base. |
e2e/cmd/k8s/e2e_test.go |
Loads YAML config (when provided), applies it to Ginkgo suite config, and runs specs. |
- Fix operator precedence: wrap each include entry in parens so entries containing && or || compose correctly with exclude labels - Fix CI path resolution: use $(CURDIR) to make config path absolute, since ginkgo runs the test binary from a different working directory - Add logs.InitLogs/FlushLogs and contextual logging to runWithTestConfig to match the setup from e2e.RunE2ETests - Add empty include label validation in the loader - Remove unimplemented Optional field from ExcludeLabel - Fix confusing error message for group name pattern validation - Add kind.yaml comment explaining why it's standalone instead of extending base.yaml (additive merge can't narrow parent includes) - Add tests for single include with || plus excludes, no-exclude case, and empty include label validation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
testconfigpackage that lets e2e pipeline configs be expressed as readable YAML files instead of long--ginkgo.focus/--ginkgo.skipregex strings. This is a first step - it converts the kind-basedmake e2e-run-testtarget to use the new config, with the intent to migrate the Semaphore pipeline files in follow-up work.The problem
Test selection in our e2e pipelines looks like this:
These strings are hard to read, hard to diff, mix labels with test-name regexes, and carry no documentation about why each pattern exists. Dead patterns accumulate silently.
The solution
A YAML config file with
include/excludesections and requiredreasonfields:Key features:
extendsfor config inheritance (e.g.,eks.yamlextendsbase.yaml,bpf-eks.yamlextendseks.yaml)reasonrequired on all exclude entries - documents why each skip existsgroupsyntax for patterns sharing a reason (e.g., all EKS DNS skips)optional: trueon exclude labels that may match 0 tests in some binaries (e.g., enterprise-only labels in OSS binary)--ginkgo.label-filter+--ginkgo.skipunder the hoodWhat changed
New
e2e/pkg/testconfig/package - types, loader (with extends resolution), converter (to ginkgo flags), and unit tests.New
e2e/config/directory with initial configs:base.yaml- shared base: sig-calico + sig-network Conformance, excludes Slow/Disruptive/ExternalNodegcp-bpf.yaml- extends base, adds VXLAN cluster exclusionskind.yaml- standalone config for kind runsMakefile -
e2e-run-testnow uses--calico.test-config=e2e/config/kind.yamlinstead ofE2E_FOCUS/E2E_SKIP.e2e/cmd/k8s/e2e_test.go- when--calico.test-configis provided, loads the config and applies label-filter/skip to the ginkgo suite config before RunSpecs.