Skip to content

Add YAML-based test selection config for e2e tests#12350

Open
caseydavenport wants to merge 2 commits intoprojectcalico:masterfrom
caseydavenport:casey-e2e-testconfig
Open

Add YAML-based test selection config for e2e tests#12350
caseydavenport wants to merge 2 commits intoprojectcalico:masterfrom
caseydavenport:casey-e2e-testconfig

Conversation

@caseydavenport
Copy link
Copy Markdown
Member

Adds a testconfig package that lets e2e pipeline configs be expressed as readable YAML files instead of long --ginkgo.focus/--ginkgo.skip regex strings. This is a first step - it converts the kind-based make e2e-run-test target to use the new config, with the intent to migrate the Semaphore pipeline files in follow-up work.

The problem

Test selection in our e2e pipelines looks like this:

--ginkgo.focus=(\[sig-calico\]|\[Conformance\]|Host-Protection) --ginkgo.skip=(\[Slow\]|\[Disruptive\]|Feature:SCTP|both.pod.and.service.Proxy\b|proxy.through.a.service.and.a.pod|...)

These strings are hard to read, hard to diff, mix labels with test-name regexes, and carry no documentation about why each pattern exists. Dead patterns accumulate silently.

The solution

A YAML config file with include/exclude sections and required reason fields:

extends: base.yaml

include:
  - sig-calico && Conformance
  - Conformance && sig-network
  - Feature:BGPPeer

exclude:
  labels:
    - label: Slow
      reason: "long-running tests not suitable for CI"
    - label: Feature:SCTP
      reason: "SCTP not supported on this platform"

  namePatterns:
    - group: "EKS control plane can't reach pod network"
      link: "https://docs.tigera.io/..."
      patterns:
        - "DNS.for.*"
        - "Proxy.version.v1"

Key features:

  • Single-parent extends for config inheritance (e.g., eks.yaml extends base.yaml, bpf-eks.yaml extends eks.yaml)
  • reason required on all exclude entries - documents why each skip exists
  • group syntax for patterns sharing a reason (e.g., all EKS DNS skips)
  • optional: true on exclude labels that may match 0 tests in some binaries (e.g., enterprise-only labels in OSS binary)
  • Circular extends detection
  • Converts to --ginkgo.label-filter + --ginkgo.skip under the hood

What changed

New e2e/pkg/testconfig/ package - types, loader (with extends resolution), converter (to ginkgo flags), and unit tests.

New e2e/config/ directory with initial configs:

  • base.yaml - shared base: sig-calico + sig-network Conformance, excludes Slow/Disruptive/ExternalNode
  • gcp-bpf.yaml - extends base, adds VXLAN cluster exclusions
  • kind.yaml - standalone config for kind runs

Makefile - e2e-run-test now uses --calico.test-config=e2e/config/kind.yaml instead of E2E_FOCUS/E2E_SKIP.

e2e/cmd/k8s/e2e_test.go - when --calico.test-config is provided, loads the config and applies label-filter/skip to the ginkgo suite config before RunSpecs.

None

Add a testconfig package that loads YAML config files defining which e2e
tests to include/exclude using Ginkgo v2 labels and test name patterns.
Config files support single-parent inheritance via 'extends', with child
configs appending to parent include/exclude lists.

The e2e test binary accepts --calico.test-config=<path> to load a config
file and apply the resulting label-filter and skip patterns to the ginkgo
suite config before running tests.

Convert the kind-based e2e-run-test Makefile target to use this instead
of the old E2E_FOCUS/E2E_SKIP make variables.
@caseydavenport caseydavenport added docs-not-required Docs not required for this change release-note-not-required Change has no user-facing impact labels Apr 4, 2026
Copilot AI review requested due to automatic review settings April 4, 2026 15:42
@caseydavenport caseydavenport added the release-note-not-required Change has no user-facing impact label Apr 4, 2026
@caseydavenport caseydavenport requested review from a team as code owners April 4, 2026 15:42
@marvin-tigera marvin-tigera added this to the Calico v3.33.0 milestone Apr 4, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a YAML-driven mechanism for selecting e2e tests (include/exclude) and wires it into the local kind-based make e2e-run-test flow, replacing hard-to-maintain ginkgo focus/skip regex strings.

Changes:

  • Added e2e/pkg/testconfig to load YAML configs (with extends inheritance) and convert them into Ginkgo v2 label-filter + skip patterns, with unit tests.
  • Added initial YAML configs under e2e/config/ (base/kind/gcp-bpf).
  • Updated the e2e test binary to accept --calico.test-config, and updated Makefile to pass the config to the e2e test run.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
Makefile Replaces E2E_FOCUS/E2E_SKIP with E2E_TEST_CONFIG and passes --calico.test-config into the e2e test binary.
e2e/pkg/testconfig/types.go Defines YAML schema/types and validation helpers for includes/excludes.
e2e/pkg/testconfig/loader.go Implements YAML loading + extends resolution + structural validation.
e2e/pkg/testconfig/convert.go Converts resolved config into Ginkgo label-filter + skip regex string(s).
e2e/pkg/testconfig/testconfig_test.go Unit tests covering loading/extends/validation and flag conversion.
e2e/pkg/config/config.go Registers --calico.test-config flag and exposes TestConfigPath().
e2e/config/base.yaml Base shared include/exclude label rules for CI-style runs.
e2e/config/kind.yaml Kind-focused selection config used by make e2e-run-test.
e2e/config/gcp-bpf.yaml GCP+BPF selection config extending base.
e2e/cmd/k8s/e2e_test.go Loads YAML config (when provided), applies it to Ginkgo suite config, and runs specs.

- Fix operator precedence: wrap each include entry in parens so entries
  containing && or || compose correctly with exclude labels
- Fix CI path resolution: use $(CURDIR) to make config path absolute,
  since ginkgo runs the test binary from a different working directory
- Add logs.InitLogs/FlushLogs and contextual logging to runWithTestConfig
  to match the setup from e2e.RunE2ETests
- Add empty include label validation in the loader
- Remove unimplemented Optional field from ExcludeLabel
- Fix confusing error message for group name pattern validation
- Add kind.yaml comment explaining why it's standalone instead of
  extending base.yaml (additive merge can't narrow parent includes)
- Add tests for single include with || plus excludes, no-exclude case,
  and empty include label validation
@dimitri-nicolo dimitri-nicolo self-requested a review April 7, 2026 18:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-not-required Docs not required for this change release-note-not-required Change has no user-facing impact

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants