Skip to content

CNTRLPLANE-2157: Migrate OTE for Pull Secrets, Feature Gates and Webhooks#31382

Open
YamunadeviShanmugam wants to merge 3 commits into
openshift:mainfrom
YamunadeviShanmugam:migrate_otp_feature_gates
Open

CNTRLPLANE-2157: Migrate OTE for Pull Secrets, Feature Gates and Webhooks#31382
YamunadeviShanmugam wants to merge 3 commits into
openshift:mainfrom
YamunadeviShanmugam:migrate_otp_feature_gates

Conversation

@YamunadeviShanmugam

@YamunadeviShanmugam YamunadeviShanmugam commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

PR_Results_FeatureGates_Webhooks_Pullsecret.txt
Adds apiserver test cases migrated to the OTE

Feature Gate tests (featuregate.go):

  • OCP-66921 — Verifies rejected/removed feature sets
  • OCP-74460 — Verifies NoUpgrade feature sets are immutable once set
  • OCP-80286 — Handle undecryptable resources: corrupts a secret in etcd, enables AllowUnsafeMalformedObjectDeletion gate, verifies kube-apiserver detects corruption, deletes via feature gate or direct etcd cleanup, confirms recovery
  • OCP-80554 — CBOR workflow: enables CBOR feature gates, verifies GET/POST/DELETE of resources using application/cbor content type

Pull Secret tests (pull_secrets.go):

  • OCP-24377 — Validates global pull secret functionality
  • OCP-12036 — Wrong credentials fail build
  • OCP-11138 — Incorrectly formed pull secrets fail deploy
  • OCP-70369 — Bound service account tokens for pull secrets

Webhook tests (webhooks.go):

  • OCP-55494 —Webhook fails to rollout latest DC
  • OCP-77919 —HPA/oc scale works with OPA webhooks
  • OCP-53230 —Kubernetes validating admission webhookbypass
  • OCP-50362 —Cluster handles bad admission webhooks correctly
  • OCP-28030 — Validates admission webhooks with unreachable services
  • OCP-50188 — APF stress test: runs kube-burner CPU stress workload for 20 minutes while monitoring cluster health, webhook behavior, and resource utilization
  • OCP-47198 — ClusterResourceQuota enforcement across namespaces

helpers.go — Shared helpers
kube-burner configs — Pod template and job config for CPU stress testing with PodSecurity privileged namespace labels

Testing
Pull secret
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-12036] User can pull a private image from a registry when a pull secret is defined [Serial][apigroup:build.openshift.io][apigroup:apps.openshift.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-11905] Use well-formed pull secret with incorrect credentials will fail to build and deploy [Serial][apigroup:build.openshift.io][apigroup:apps.openshift.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-11138] Deploy will fail with incorrectly formed pull secrets [apigroup:build.openshift.io][apigroup:apps.openshift.io][apigroup:image.openshift.io] [Suite:openshift/conformance/parallel]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-70369] Use bound service account tokens when generating pull secrets [apigroup:image.openshift.io] [Suite:openshift/conformance/parallel]"

Feature Gates
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-66921-1] should reject removed LatencySensitive featureset [apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-66921-2][OCP-74460] NoUpgrade featuresets are immutable once set [Slow][Disruptive][apigroup:config.openshift.io][Timeout:30m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-80286] Handle undecryptable resources [Slow][Disruptive][apigroup:config.openshift.io][apigroup:operator.openshift.io][Timeout:90m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-80554] Verify the CBOR workflow [Slow][Disruptive][apigroup:config.openshift.io][Timeout:60m] [Serial]"

Webhooks
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-55494] When using webhooks fails to rollout latest deploymentconfig [Disruptive][apigroup:apps.openshift.io] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-77919] HPA/oc scale and DeploymentConfig should be working with OPA webhooks [Disruptive][apigroup:apps.openshift.io] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-53230] Kubernetes validating admission webhook bypass [Serial][apigroup:admissionregistration.k8s.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50362] Verify cluster handles bad admission webhooks correctly [Serial][apigroup:config.openshift.io][apigroup:admissionregistration.k8s.io][apigroup:apiextensions.k8s.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50223] Checks on different bad admission webhook errors and kube-apiserver status [Serial][apigroup:admissionregistration.k8s.io][apigroup:apiextensions.k8s.io][Timeout:15m] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50188] Prepare upgrade cluster under APF stress [Slow][Disruptive][apigroup:config.openshift.io][Timeout:40m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP] ClusterResourceQuota objects validation [apigroup:quota.openshift.io][apigroup:monitoring.coreos.com] [Suite:openshift/conformance/parallel]"

Summary by CodeRabbit

  • New Features

    • Added comprehensive API server end-to-end coverage for feature gates, CBOR requests, pull secrets, admission webhooks, upgrades, quotas, and corrupted resources.
    • Added validation for webhook admission behavior, image-pull failures, service account token reconciliation, and API server recovery.
    • Added CPU stress workload fixtures for performance and stability testing.
  • Tests

    • Added health checks, operator-state validation, workload monitoring, and cluster recovery assertions across new scenarios.

Signed-off-by: Yamunadevi Shanmugam <yshanmug@redhat.com>
Signed-off-by: Yamunadevi Shanmugam <yshanmug@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot requested review from p0lyn0mial and rexagod July 14, 2026 04:43
@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: YamunadeviShanmugam
Once this PR has been reviewed and has the lgtm label, please assign miyadav for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Walkthrough

Adds extensive OpenShift API server extended tests covering feature gates, CBOR operations, image pull secrets, admission webhooks, operator recovery, ClusterResourceQuota enforcement, and APF stress, supported by shared helpers and embedded kube-burner fixtures.

Changes

API server extended test coverage

Layer / File(s) Summary
Shared test and cluster utilities
test/extended/apiserver/helpers.go
Adds command, polling, proxy, certificate, operator, cluster health, pod, and resource utility functions for API server tests.
Feature, webhook, quota, and workload support
test/extended/apiserver/helpers.go
Adds feature-gate, admission webhook, quota, cluster-load, resource-count, and kube-burner orchestration helpers.
Feature-gate and CBOR workflows
go.mod, test/extended/apiserver/featuregate.go
Adds OTP coverage for invalid and immutable feature sets, corrupted objects, and CBOR GET, POST, and DELETE operations; declares CBOR as a direct dependency.
Pull-secret behavior tests
test/extended/apiserver/pull_secrets.go
Tests valid, invalid, missing, mismatched, and bound-service-account image pull-secret behavior.
Webhook, upgrade, quota, and stress tests
test/extended/apiserver/webhooks.go, test/extended/testdata/apiserver/*, test/extended/testdata/bindata.go
Adds admission webhook, malformed webhook upgrade, APF stress, and ClusterResourceQuota tests with embedded kube-burner fixtures.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Test as Extended API server test
  participant KubeAPIServer as kube-apiserver
  participant OPA as OPA webhook
  participant Resource as DeploymentConfig or node resource
  Test->>OPA: setupOPAWebhook and apply policy
  Test->>KubeAPIServer: submit resource operation
  KubeAPIServer->>OPA: admission review
  OPA-->>KubeAPIServer: allow or deny decision
  KubeAPIServer->>Resource: apply accepted operation
  Test->>KubeAPIServer: verify operator conditions and resource state
Loading

Suggested reviewers: p0lyn0mial, rexagod


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 3 warnings)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error helpers.go logs raw HTTP/proxy errors; those can include internal hostnames or proxy URLs in e2e output. Sanitize these logs: remove raw err text or mask host/URL details before logging, especially in urlHealthCheck and proxy parsing.
Test Structure And Quality ⚠️ Warning FAIL: featuregate 66921-2 mutates cluster featuregates with no restore/AfterEach, and many new assertions are bare Expect(err).NotTo(HaveOccurred()) without context. Add cleanup/restore for cluster-scoped mutations (feature gates, roles, namespaces) and attach contextual messages to resource-creation/wait assertions.
Microshift Test Compatibility ⚠️ Warning 50223 uses kube-apiserver ClusterOperator checks without a MicroShift-skip tag, and 53230 requires a master node via GetFirstMasterNode(). Add a MicroShift skip or an unavailable-apigroup tag for 50223, and guard 53230 with a MicroShift check or avoid master-node assumptions.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning FAIL: new pull-secret tests use quay.io and github.com, and webhooks hardcode 127.0.0.1 in cert SAN; both break disconnected/IPv6-only compatibility. Use internal/mirrored images or cluster-local sources instead of quay.io/github.com, and remove IPv4 localhost assumptions (or add IPv6-aware branching) then validate in the IPv6 disconnected CI job.
✅ Passed checks (11 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All new Ginkgo titles are static literals; randomness and time-based values stay in test bodies, not It/Describe names.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new test assumes multiple nodes/HA: the suites only use cluster APIs, pod readiness, and per-role health checks; no SNO guard was needed.
Topology-Aware Scheduling Compatibility ✅ Passed No added nodeSelectors, affinity, tolerations, PDBs, or topology-derived replicas; the new stress pod/job are unpinned and helpers only observe node roles.
Ote Binary Stdout Contract ✅ Passed No stdout writes were added in init/suite setup/top-level initializers; the only fmt.Printf is inside an It block, which is allowed.
No-Weak-Crypto ✅ Passed No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token non-constant-time compares were added in the changed files.
Container-Privileges ✅ Passed No added diff lines set privileged/hostPID/hostNetwork/hostIPC, SYS_ADMIN, runAsUser: 0, or allowPrivilegeEscalation; the new pod template has no securityContext.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: migrating OTE coverage for pull secrets, feature gates, and webhooks.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 15

🧹 Nitpick comments (6)
test/extended/prometheus/upgrade.go (1)

118-123: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Simplify substring counting and avoid searching YAML metadata.

Using a regular expression to match a static string is inefficient, and printing the resulting slice of identical strings (e.g., [failed failed]) is not helpful for debugging. Furthermore, searching the full -o yaml output risks matching the word failed in annotations, labels, or other metadata, leading to false positives.

Consider outputting only the .data payload and using strings.Count for a simpler, clearer, and more robust check. As per coding guidelines, favor clarity and maintainability over cleverness.

♻️ Proposed refactor for clarity and accuracy
-		result, err := oc.AsAdmin().WithoutNamespace().Run("get").Args("cm/log", "-n", "ocp-34223-proj", "-o", "yaml").Output()
+		result, err := oc.AsAdmin().WithoutNamespace().Run("get").Args("cm/log", "-n", "ocp-34223-proj", "-o", "jsonpath={.data}").Output()
 		o.Expect(err).NotTo(o.HaveOccurred())
-		failures := regexp.MustCompile(`failed`).FindAllString(result, -1)
-		if len(failures) > 1 {
-			e2e.Failf("upgrade disruption detected in monitor log: %v", failures)
+		failuresCount := strings.Count(result, "failed")
+		if failuresCount > 1 {
+			e2e.Failf("upgrade disruption detected in monitor log: found 'failed' %d times", failuresCount)
 		}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/prometheus/upgrade.go` around lines 118 - 123, Update the
Prometheus upgrade check around the oc.AsAdmin().WithoutNamespace().Run call to
request only the ConfigMap’s .data payload instead of full YAML, then replace
the regexp-based failure search with strings.Count. Preserve the existing
threshold of more than one occurrence, and report the count in e2e.Failf rather
than a slice of repeated matches.

Source: Coding guidelines

test/extended/apiserver/helpers.go (2)

271-271: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Use net.JoinHostPort instead of fmt.Sprintf for host:port.

Raw %s:%s breaks for IPv6 hostnames (missing bracket notation); net.JoinHostPort handles this correctly.

-targetURL := fmt.Sprintf("https://%s:%s/healthz", fqdnName, port)
+targetURL := fmt.Sprintf("https://%s/healthz", net.JoinHostPort(fqdnName, port))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/apiserver/helpers.go` at line 271, Update the targetURL
construction to use net.JoinHostPort with fqdnName and port, preserving the
/healthz suffix so host:port formatting remains correct for IPv4, hostnames, and
IPv6 addresses.

Source: Linters/SAST tools


264-269: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Set an explicit MinVersion on the TLS client config.

Go's client-side default minimum is currently TLS 1.2, but pinning it explicitly (ideally TLS 1.3) documents intent and protects against future default changes.

 TLSClientConfig: &tls.Config{
     RootCAs: caCertPool,
+    MinVersion: tls.VersionTLS13,
 },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/apiserver/helpers.go` around lines 264 - 269, Update the TLS
client configuration in the HTTP transport to set an explicit MinVersion,
preferably tls.VersionTLS13, while preserving the existing RootCAs
configuration.

Source: Linters/SAST tools

test/extended/testdata/apiserver/kube-burner-cpu-stress-pod.yml (1)

8-21: 🔒 Security & Privacy | 🔵 Trivial

No explicit securityContext on the stress container.

Checkov flags missing allowPrivilegeEscalation: false and root-container admission. In practice this is mitigated since loadCPUMemWorkload (helpers.go) explicitly labels the target namespace pod-security.kubernetes.io/enforce=privileged before applying this template, so it's not a functional blocker — but adding an explicit securityContext would still be good defense-in-depth per the path instructions for Kubernetes manifests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/testdata/apiserver/kube-burner-cpu-stress-pod.yml` around lines
8 - 21, Add an explicit securityContext to the stress container in
kube-burner-cpu-stress-pod.yml, setting allowPrivilegeEscalation to false and
configuring the container to run as a non-root user where compatible with the
stress image. Keep the existing command, resources, and restart policy
unchanged.

Sources: Path instructions, Linters/SAST tools

test/extended/apiserver/webhooks.go (2)

146-240: 📐 Maintainability & Code Quality | 🔵 Trivial

Duplicated namespace/DC-creation/rollout-wait flow between OCP-55494 and OCP-77919.

Both g.It blocks repeat the same "create test-ns, label it, create mydc, wait for rollout" sequence almost verbatim (159-178 vs 213-232). Could be extracted into a shared helper to reduce duplication, though this is purely a maintainability nit for test code.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/apiserver/webhooks.go` around lines 146 - 240, Extract the
duplicated namespace creation, labeling, DeploymentConfig creation, and initial
rollout-wait sequence from the OCP-55494 and OCP-77919 g.It blocks into a shared
helper. Replace both inline flows with calls to that helper, preserving their
existing namespace names, assertions, polling behavior, and error messages.

66-98: 🔒 Security & Privacy | 🔵 Trivial

Avoid bash -c shell wrapper for exec.Command.

Static analysis (OpenGrep/ast-grep) flags every exec.Command("bash", "-c", <dynamic string>) call in this file as a command-injection risk. None of the interpolated values here are attacker-controlled (they're tmpdir paths built from os.TempDir()+compat_otp.GetRandomString(), or fixed constants like dirname/exceptions), so this isn't currently exploitable, but it's a recurring anti-pattern the path instructions call out explicitly for command execution. Prefer invoking binaries directly (e.g. exec.Command("openssl", "genrsa", "-out", caKeypem, "2048")) instead of shelling out, which also sidesteps future risk if any of these values are ever derived from less-trusted input.

As per path instructions, **/*.{py,js,ts,go,rs,java,rb,php,kt,swift,cs}: "Command: no shell=True, os.system, or backtick exec with user input."

Also applies to: 120-122, 181-190, 535-536, 554-555

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/apiserver/webhooks.go` around lines 66 - 98, Replace the
dynamic bash -c invocations in the webhook certificate setup and the other
flagged command-execution sites with direct exec.Command calls, passing the
executable and each argument separately. Update the command loops and related
symbols such as serverconfCMD so no shell command strings are constructed or
interpreted, while preserving the existing OpenSSL and file-generation behavior.

Sources: Path instructions, Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/apiserver/featuregate.go`:
- Around line 190-215: Extract the duplicated feature-gate restoration logic
from both deferred restore blocks into a shared restoreFeatureGateConfig helper,
placed with the existing helpers. Have it accept the CLI and original feature
settings, build and apply the same patch, and return any patch error; update
both tests to call it while preserving their existing warning and kube-apiserver
stabilization behavior.
- Around line 234-286: Extract the duplicated customNoUpgrade merge, patch, and
kube-apiserver rollout logic from the two tests into a shared enableFeatureGates
helper accepting oc and gatesToAdd and returning whether all requested gates
were already enabled plus an error. Update both call sites to use the helper,
preserving existing feature-set selection, no-change handling, rollout waits,
and health verification while differing only in the gate names passed.
- Around line 535-539: Update the deletion poll assigned to errDel so it
verifies that the pod get operation returns a NotFound/error condition, matching
the existing pollUntilDeleted pattern, instead of checking table output for the
quoted “Running” status. Keep polling while the pod exists and return success
only after test-pod-nginx has been deleted, preserving
compat_otp.AssertWaitPollNoErr for reporting failures.
- Around line 412-428: The kubeconfig extraction block should use
client-go/clientcmd parsing instead of grep/awk/base64 pipelines. Replace the
dead clusterName lookup and shell commands with loading the selected kubeconfig
context, retrieving its cluster CA and user client certificate/key data, and
writing those bytes directly to certCA, clientKey, and clientCert while
preserving the existing error assertions.

In `@test/extended/apiserver/helpers.go`:
- Around line 469-489: Update getServiceIP so the all-namespaces service
cluster-IP query runs once before the lastOctet loop; retain its error
assertion, then check the returned service IP list against each candidate in
memory while preserving the existing fallback when no address is available.
- Around line 273-297: Update the polling callback around client.Get to create
an HTTP request using the provided ctx and execute it through the existing
client, ensuring each attempt is canceled when the poll context expires.
Configure an appropriate client-level timeout as an additional bound, while
preserving the existing retry and certificate-processing behavior.
- Around line 599-611: Update checkResources to count output rows rather than
whitespace-separated fields: after trimming each successful resource output,
count its lines and store that value in counts[resource]. Preserve the existing
handling for command errors and empty output.
- Around line 566-597: Update checkClusterLoad so memory parsing uses the second
percentage on each node line rather than the first CPU percentage; ensure
memTotal reflects MEMORY% while CPU parsing remains unchanged. Use the existing
cpuMatches and memMatches flow, or otherwise parse the output columns
explicitly.
- Around line 613-675: Update loadCPUMemWorkload to launch kube-burner with
exec.CommandContext and a bounded timeout, creating and cancelling the context
around the child process execution. Preserve the existing arguments,
CombinedOutput handling, and skip message while ensuring a stuck process is
terminated after the timeout.

In `@test/extended/apiserver/webhooks.go`:
- Around line 108-110: Update setupOPAWebhook and the corresponding second g.It
block to revoke the privileged SCC grant during deferred cleanup. Add cleanup
that removes the system:serviceaccount:opa:default subject from the privileged
SCC binding using the matching oc adm policy command, while preserving the
existing namespace and opa-viewer cleanup.
- Around line 520-543: Ensure the APF stress scenario does not continue when the
pre-check in checkClusterLoad prevents loadCPUMemWorkload from starting. Track
whether the stress workload was launched, then fail fast or skip before the
verification poll when it was not started; retain the existing cleanup defer
only for a started workload and keep the normal pod-completion and health checks
for the active workload path.
- Around line 601-652: The quota-boundary checks currently reuse the initial
pods and resource usage snapshot despite creating a ServiceMonitor beforehand.
Re-query the cluster resource quota status immediately before the ServiceMonitor
and pod boundary loops, refresh the relevant usage values (including pods and
ServiceMonitors), and use those refreshed counts to determine expected successes
and quota-exceeded cases.

In `@test/extended/prometheus/upgrade.go`:
- Around line 114-116: Update the ConfigMap existence check around cmExistsCmd
and err so the test skips only when the command reports a resource-not-found
condition, including the expected “NotFound” response for a specific cm/log
query. Do not skip for arbitrary command execution errors; propagate or fail
those errors through the existing test error-handling path.
- Line 111: Handle the error returned by Execute() in the deferred namespace
cleanup for ocp-34223-proj. Update the defer logic around
AsAdmin().WithoutNamespace().Run("delete") so cleanup failures are explicitly
checked and reported, without silently discarding the error.

In `@test/extended/testdata/apiserver/kube-burner-cpu-stress.yml`:
- Around line 11-13: Regenerate the embedded assets represented by
test/extended/testdata/bindata.go from the corrected
test/extended/testdata/apiserver/kube-burner-cpu-stress.yml, using the
repository’s standard generation command such as make update. Ensure
_testExtendedTestdataApiserverKubeBurnerCpuStressYml matches podWait: true and
cleanup: false; both listed sites require synchronization, with no manual
divergence between the source fixture and compiled asset.

---

Nitpick comments:
In `@test/extended/apiserver/helpers.go`:
- Line 271: Update the targetURL construction to use net.JoinHostPort with
fqdnName and port, preserving the /healthz suffix so host:port formatting
remains correct for IPv4, hostnames, and IPv6 addresses.
- Around line 264-269: Update the TLS client configuration in the HTTP transport
to set an explicit MinVersion, preferably tls.VersionTLS13, while preserving the
existing RootCAs configuration.

In `@test/extended/apiserver/webhooks.go`:
- Around line 146-240: Extract the duplicated namespace creation, labeling,
DeploymentConfig creation, and initial rollout-wait sequence from the OCP-55494
and OCP-77919 g.It blocks into a shared helper. Replace both inline flows with
calls to that helper, preserving their existing namespace names, assertions,
polling behavior, and error messages.
- Around line 66-98: Replace the dynamic bash -c invocations in the webhook
certificate setup and the other flagged command-execution sites with direct
exec.Command calls, passing the executable and each argument separately. Update
the command loops and related symbols such as serverconfCMD so no shell command
strings are constructed or interpreted, while preserving the existing OpenSSL
and file-generation behavior.

In `@test/extended/prometheus/upgrade.go`:
- Around line 118-123: Update the Prometheus upgrade check around the
oc.AsAdmin().WithoutNamespace().Run call to request only the ConfigMap’s .data
payload instead of full YAML, then replace the regexp-based failure search with
strings.Count. Preserve the existing threshold of more than one occurrence, and
report the count in e2e.Failf rather than a slice of repeated matches.

In `@test/extended/testdata/apiserver/kube-burner-cpu-stress-pod.yml`:
- Around line 8-21: Add an explicit securityContext to the stress container in
kube-burner-cpu-stress-pod.yml, setting allowPrivilegeEscalation to false and
configuring the container to run as a non-root user where compatible with the
stress image. Keep the existing command, resources, and restart policy
unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 9832f67a-fdf7-45b3-afb3-bb006aa05ecc

📥 Commits

Reviewing files that changed from the base of the PR and between 0f64c49 and 6722b7e.

📒 Files selected for processing (9)
  • go.mod
  • test/extended/apiserver/featuregate.go
  • test/extended/apiserver/helpers.go
  • test/extended/apiserver/pull_secrets.go
  • test/extended/apiserver/webhooks.go
  • test/extended/prometheus/upgrade.go
  • test/extended/testdata/apiserver/kube-burner-cpu-stress-pod.yml
  • test/extended/testdata/apiserver/kube-burner-cpu-stress.yml
  • test/extended/testdata/bindata.go

Comment thread test/extended/apiserver/featuregate.go
Comment thread test/extended/apiserver/featuregate.go Outdated
Comment thread test/extended/apiserver/featuregate.go Outdated
Comment thread test/extended/apiserver/featuregate.go
Comment thread test/extended/apiserver/helpers.go
Comment thread test/extended/apiserver/webhooks.go
Comment thread test/extended/apiserver/webhooks.go
Comment thread test/extended/prometheus/upgrade.go Outdated
oc := exutil.NewCLIWithoutNamespace("apiserver-zero-disruption-upgrade")

g.It("kube-apiserver and openshift-apiserver should have zero-disruption upgrade [apigroup:config.openshift.io]", func() {
defer oc.AsAdmin().WithoutNamespace().Run("delete").Args("ns", "ocp-34223-proj", "--ignore-not-found").Execute()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Handle the error returned by Execute().

Ignoring the error from the namespace deletion cleanup could mask teardown issues. As per path instructions for Go security, you must never ignore error returns.

🛠️ Proposed fix to handle the cleanup error
-		defer oc.AsAdmin().WithoutNamespace().Run("delete").Args("ns", "ocp-34223-proj", "--ignore-not-found").Execute()
+		defer func() {
+			if err := oc.AsAdmin().WithoutNamespace().Run("delete").Args("ns", "ocp-34223-proj", "--ignore-not-found").Execute(); err != nil {
+				e2e.Logf("Failed to delete namespace ocp-34223-proj: %v", err)
+			}
+		}()
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
defer oc.AsAdmin().WithoutNamespace().Run("delete").Args("ns", "ocp-34223-proj", "--ignore-not-found").Execute()
defer func() {
if err := oc.AsAdmin().WithoutNamespace().Run("delete").Args("ns", "ocp-34223-proj", "--ignore-not-found").Execute(); err != nil {
e2e.Logf("Failed to delete namespace ocp-34223-proj: %v", err)
}
}()
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/prometheus/upgrade.go` at line 111, Handle the error returned
by Execute() in the deferred namespace cleanup for ocp-34223-proj. Update the
defer logic around AsAdmin().WithoutNamespace().Run("delete") so cleanup
failures are explicitly checked and reported, without silently discarding the
error.

Source: Path instructions

Comment thread test/extended/prometheus/upgrade.go Outdated
Comment on lines +114 to +116
if strings.Contains(cmExistsCmd, "No resources found") || err != nil {
g.Skip("ConfigMap log in ocp-34223-proj does not exist; pre-upgrade monitor job was not run")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not unconditionally skip the test on any error.

The || err != nil condition will skip the test on any command execution error (e.g., API server unreachable, network timeouts), masking real cluster or infrastructure failures. It should specifically check for the NotFound error. Additionally, querying a specific resource name like cm/log typically outputs NotFound rather than No resources found.

🛡️ Proposed fix to strictly check for not-found errors
-		if strings.Contains(cmExistsCmd, "No resources found") || err != nil {
-			g.Skip("ConfigMap log in ocp-34223-proj does not exist; pre-upgrade monitor job was not run")
-		}
+		if err != nil {
+			if strings.Contains(cmExistsCmd, "NotFound") || strings.Contains(cmExistsCmd, "No resources found") {
+				g.Skip("ConfigMap log in ocp-34223-proj does not exist; pre-upgrade monitor job was not run")
+			}
+			o.Expect(err).NotTo(o.HaveOccurred())
+		}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if strings.Contains(cmExistsCmd, "No resources found") || err != nil {
g.Skip("ConfigMap log in ocp-34223-proj does not exist; pre-upgrade monitor job was not run")
}
if err != nil {
if strings.Contains(cmExistsCmd, "NotFound") || strings.Contains(cmExistsCmd, "No resources found") {
g.Skip("ConfigMap log in ocp-34223-proj does not exist; pre-upgrade monitor job was not run")
}
o.Expect(err).NotTo(o.HaveOccurred())
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/prometheus/upgrade.go` around lines 114 - 116, Update the
ConfigMap existence check around cmExistsCmd and err so the test skips only when
the command reports a resource-not-found condition, including the expected
“NotFound” response for a specific cm/log query. Do not skip for arbitrary
command execution errors; propagate or fail those errors through the existing
test error-handling path.

Comment thread test/extended/testdata/apiserver/kube-burner-cpu-stress.yml
Signed-off-by: Yamunadevi Shanmugam <yshanmug@redhat.com>
@YamunadeviShanmugam YamunadeviShanmugam force-pushed the migrate_otp_feature_gates branch from 2b72d09 to ab3de70 Compare July 14, 2026 05:45
@YamunadeviShanmugam

YamunadeviShanmugam commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

/retitle CNTRLPLANE-2157: Migrate OTE for Pull Secrets, Feature Gates and Webhooks

@openshift-ci openshift-ci Bot changed the title Migrate OTE for Pull Secrets, Feature Gates and Webhooks CNTRLPLANE-2157: Migrate OTE for Pull Secrets, Feature Gates and Webhooks Jul 14, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 14, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 14, 2026

Copy link
Copy Markdown

@YamunadeviShanmugam: This pull request references CNTRLPLANE-2157 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Adds apiserver test cases migrated to the OTE

Feature Gate tests (featuregate.go):

  • OCP-66921 — Verifies rejected/removed feature sets
  • OCP-74460 — Verifies NoUpgrade feature sets are immutable once set
  • OCP-80286 — Handle undecryptable resources: corrupts a secret in etcd, enables AllowUnsafeMalformedObjectDeletion gate, verifies kube-apiserver detects corruption, deletes via feature gate or direct etcd cleanup, confirms recovery
  • OCP-80554 — CBOR workflow: enables CBOR feature gates, verifies GET/POST/DELETE of resources using application/cbor content type

Pull Secret tests (pull_secrets.go):

  • OCP-24377 — Validates global pull secret functionality
  • OCP-12036 — Wrong credentials fail build
  • OCP-11138 — Incorrectly formed pull secrets fail deploy
  • OCP-70369 — Bound service account tokens for pull secrets

Webhook tests (webhooks.go):

  • OCP-55494 —Webhook fails to rollout latest DC
  • OCP-77919 —HPA/oc scale works with OPA webhooks
  • OCP-53230 —Kubernetes validating admission webhookbypass
  • OCP-50362 —Cluster handles bad admission webhooks correctly
  • OCP-28030 — Validates admission webhooks with unreachable services
  • OCP-50188 — APF stress test: runs kube-burner CPU stress workload for 20 minutes while monitoring cluster health, webhook behavior, and resource utilization
  • OCP-47198 — ClusterResourceQuota enforcement across namespaces

helpers.go — Shared helpers
kube-burner configs — Pod template and job config for CPU stress testing with PodSecurity privileged namespace labels

Testing
Pull secret
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-12036] User can pull a private image from a registry when a pull secret is defined [Serial][apigroup:build.openshift.io][apigroup:apps.openshift.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-11905] Use well-formed pull secret with incorrect credentials will fail to build and deploy [Serial][apigroup:build.openshift.io][apigroup:apps.openshift.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-11138] Deploy will fail with incorrectly formed pull secrets [apigroup:build.openshift.io][apigroup:apps.openshift.io][apigroup:image.openshift.io] [Suite:openshift/conformance/parallel]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:PullSecret] [OTP][OCP-70369] Use bound service account tokens when generating pull secrets [apigroup:image.openshift.io] [Suite:openshift/conformance/parallel]"

Feature Gates
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-66921-1] should reject removed LatencySensitive featureset [apigroup:config.openshift.io] [Suite:openshift/conformance/parallel]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-66921-2][OCP-74460] NoUpgrade featuresets are immutable once set [Slow][Disruptive][apigroup:config.openshift.io][Timeout:30m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-80286] Handle undecryptable resources [Slow][Disruptive][apigroup:config.openshift.io][apigroup:operator.openshift.io][Timeout:90m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:FeatureGate] [OTP][OCP-80554] Verify the CBOR workflow [Slow][Disruptive][apigroup:config.openshift.io][Timeout:60m] [Serial]"

Webhooks
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-55494] When using webhooks fails to rollout latest deploymentconfig [Disruptive][apigroup:apps.openshift.io] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-77919] HPA/oc scale and DeploymentConfig should be working with OPA webhooks [Disruptive][apigroup:apps.openshift.io] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:Webhooks] [OTP][OCP-53230] Kubernetes validating admission webhook bypass [Serial][apigroup:admissionregistration.k8s.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50362] Verify cluster handles bad admission webhooks correctly [Serial][apigroup:config.openshift.io][apigroup:admissionregistration.k8s.io][apigroup:apiextensions.k8s.io] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50223] Checks on different bad admission webhook errors and kube-apiserver status [Serial][apigroup:admissionregistration.k8s.io][apigroup:apiextensions.k8s.io][Timeout:15m] [Suite:openshift/conformance/serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP][OCP-50188] Prepare upgrade cluster under APF stress [Slow][Disruptive][apigroup:config.openshift.io][Timeout:40m] [Serial]"
./openshift-tests run-test "[sig-api-machinery][Feature:APIServer][Feature:UpgradeWebhooks] [OTP] ClusterResourceQuota objects validation [apigroup:quota.openshift.io][apigroup:monitoring.coreos.com] [Suite:openshift/conformance/parallel]"

Summary by CodeRabbit

  • New Features

  • Added comprehensive API server end-to-end coverage for feature gates, CBOR requests, pull secrets, admission webhooks, upgrades, quotas, and corrupted resources.

  • Added validation for webhook admission behavior, image-pull failures, service account token reconciliation, and API server recovery.

  • Added CPU stress workload fixtures for performance and stability testing.

  • Tests

  • Added health checks, operator-state validation, workload monitoring, and cluster recovery assertions across new scenarios.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

@YamunadeviShanmugam: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify ab3de70 link true /test verify
ci/prow/go-verify-deps ab3de70 link true /test go-verify-deps

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants