Skip to content

OPRUN-3863: Add olmv1 webhook support origin tests#29825

Closed
perdasilva wants to merge 1 commit intoopenshift:mainfrom
perdasilva:olmv1-webhook-tests
Closed

OPRUN-3863: Add olmv1 webhook support origin tests#29825
perdasilva wants to merge 1 commit intoopenshift:mainfrom
perdasilva:olmv1-webhook-tests

Conversation

@perdasilva
Copy link

@perdasilva perdasilva commented May 20, 2025

Adds origin tests for the webhook support feature. Test need to be run serially to enable BeforeSuite functionality which installs the webhook operator for the tests:

  • check validating webhook
  • check mutating webhook
  • check conversion webhook
  • check service availability during cert secret deletion by continuously creating and deleting a resource after the deletion of the cert secret
  • check deployment availability during cert rotation, which is forced to happen when the openshift-service-ca signing key gets deleted.

(I'm not sure I'd add the last two if it weren't for the need to have 5 tests)

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 20, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented May 20, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from 1816e5d to c7e942a Compare May 20, 2025 14:16
@perdasilva
Copy link
Author

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

@perdasilva perdasilva changed the title Add olmv1 webhook support origin tests OPRUN-3863: Add olmv1 webhook support origin tests May 20, 2025
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 20, 2025
@openshift-ci-robot
Copy link

openshift-ci-robot commented May 20, 2025

@perdasilva: This pull request references OPRUN-3863 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch 4 times, most recently from 2a43c3a to e1e5feb Compare May 26, 2025 15:25
@perdasilva
Copy link
Author

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

@openshift-trt
Copy link

openshift-trt bot commented May 26, 2025

Job Failure Risk Analysis for sha: e1e5feb

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-dualstack-bgp-techpreview IncompleteTests
Tests for this run (22) are below the historical average (3048): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch 2 times, most recently from d3c2644 to 44a9c22 Compare May 27, 2025 13:10
@perdasilva perdasilva marked this pull request as ready for review May 27, 2025 13:11
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 27, 2025
@perdasilva
Copy link
Author

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

@openshift-ci openshift-ci bot requested review from ankitathomas and orenc1 May 27, 2025 13:13
@openshift-ci-robot
Copy link

openshift-ci-robot commented May 27, 2025

@perdasilva: This pull request references OPRUN-3863 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.20.0" version, but no target version was set.

Details

In response to this:

Adds origin tests for the webhook support feature. Test need to be run serially to enable BeforeSuite functionality which installs the webhook operator for the tests:

  • check validating webhook
  • check mutating webhook
  • check conversion webhook
  • check tls secret exists
  • check webhook service exists with openshift-serviceca annotation

(I'm not sure I'd add the last two if it weren't for the need to have 5 tests)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@perdasilva
Copy link
Author

/retest

@openshift-trt
Copy link

openshift-trt bot commented May 30, 2025

Job Failure Risk Analysis for sha: 44a9c22

Job Name Failure Risk
pull-ci-openshift-origin-main-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback IncompleteTests
Tests for this run (12) are below the historical average (245): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-gcp-disruptive Medium
[sig-node] static pods should start after being created
Potential external regression detected for High Risk Test analysis
---
[sig-arch][Late] operators should not create watch channels very often
Potential external regression detected for High Risk Test analysis

Open Bugs
Component Readiness Shows Old Test Name For Renamed Tests
ResilientWatchCacheInitialization (Re)enablement - operator watch counts from component readiness
pull-ci-openshift-origin-main-e2e-gcp-ovn-etcd-scaling Low
[bz-kube-storage-version-migrator] clusteroperator/kube-storage-version-migrator should not change condition/Available
This test has passed 69.42% of 5144 runs on release 4.20 [Overall] in the last week.
---
[bz-Cloud Compute] clusteroperator/control-plane-machine-set should not change condition/Degraded
This test has passed 0.00% of 1 runs on release 4.20 [Architecture:amd64 FeatureSet:default Installer:ipi JobTier:rare Network:ovn NetworkStack:ipv4 Owner:eng Platform:gcp SecurityMode:default Topology:ha Upgrade:none] in the last week.

certificateSecretName = "webhook-operator-webhook-service-cert"
)

var _ = g.Describe("[sig-olmv1][OCPFeatureGate:NewOLMWebhookProviderOpenshiftServiceCA][Skipped:Disconnected][Serial] OLMv1 operator with webhooks", g.Ordered, g.Serial, func() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there is some way to test certificate rotation and check that the service is available after a rotation occurs? I feel like that has been problematic in OLMv0's implementation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a great question, and I'm really not sure how much we can control openshift-serviceca's rotation period. Let me poke at it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the two tests:

  • ensuring service availability during cert secret deletion by continuously creating and deleting a resource after the deletion of the cert secret
  • ensuring deployment availability during cert rotation, which is forced to happen when the openshift-service-ca signing key gets deleted. The problem I ran into is that then the certificate is no longer signed and fails for another reason. So, I'm not sure the best way to test this case =S maybe we just cannot...

Copy link
Member

@joelanford joelanford Jul 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the latter case, I think my expectation would be:

  1. If service-ca signing key is deleted, then service-ca-operator creates a new signing key and then rotates all certs.
  2. When service-ca-operator creates a new signing key, the webhook configurations are updated to reflect the new key and the kubernetes API server starts using that key to validate the webhook servers.
  3. When service-ca-operator rotates the webhook server's cert (so that it is signed by the new signing key), the server pods notice that change and start using that new cert.

2 and 3 wouldn't happen atomically at the same time, so I think it would be safe to assume some downtime, between catalogd and operator-controller, but then for that to correct itself on its own after both (2) and (3) occur.

Is that your understanding of what should happen?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's reasonable. I was wondering if it would be a controlled hand-off to the new pod by the deployment.
I.e. when the tls secret changes, deployment notices, rotates the pods by bringing up the new, switching traffic, then bringing down the old one. But, it doesn't sound unreasonable that there might be some downtime.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The service-ca validity period is 26 months, a time period where users are expected to have upgraded, such that the certificate is rotated as part of an upgrade.

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from 44a9c22 to 51a650d Compare June 4, 2025 11:11
@openshift-trt
Copy link

openshift-trt bot commented Jun 4, 2025

Job Failure Risk Analysis for sha: 51a650d

Job Name Failure Risk
pull-ci-openshift-origin-main-4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback IncompleteTests
Tests for this run (86) are below the historical average (231): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-aws-disruptive Medium
[bz-openshift-apiserver] clusteroperator/openshift-apiserver should not change condition/Available
Potential external regression detected for High Risk Test analysis

Open Bugs
CI: API is broken in periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-single-node-techpreview-serial
openshift-apiserver ClusterOperator should not blip Available=False on brief missing HTTP content-type
---
[sig-network] pods should successfully create sandboxes by writing network status
Potential external regression detected for High Risk Test analysis

Open Bugs
etcd timeouts causing failed pod sandbox creation writing network status

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from 51a650d to 36008d8 Compare June 5, 2025 10:06
@perdasilva
Copy link
Author

/test e2e-metal-ipi-ovn-dualstack-bgp-techpreview

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from c7bb08c to e9b29a9 Compare July 11, 2025 12:20
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 11, 2025
@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from e9b29a9 to d88423b Compare July 11, 2025 13:12
@perdasilva
Copy link
Author

/payload-job periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 14, 2025

@perdasilva: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/bde39b50-60b6-11f0-8574-1255cecb9182-0

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from d88423b to c9e6602 Compare July 14, 2025 14:43
@perdasilva
Copy link
Author

perdasilva commented Jul 14, 2025

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 14, 2025

@perdasilva: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6a587380-60c9-11f0-9e53-2a134413be2d-0

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 14, 2025

@perdasilva: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/22d2d510-60cd-11f0-9076-ff16c0ab5472-0

@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from c9e6602 to f8a8aa0 Compare July 21, 2025 11:24
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 28, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: perdasilva, thetechnick
Once this PR has been reviewed and has the lgtm label, please assign xueqzhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2025
@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from f8a8aa0 to 15aa5f8 Compare July 28, 2025 16:27
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 28, 2025
Signed-off-by: Per Goncalves da Silva <pegoncal@redhat.com>
@perdasilva perdasilva force-pushed the olmv1-webhook-tests branch from 15aa5f8 to cce8714 Compare July 29, 2025 09:43
@perdasilva
Copy link
Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 29, 2025

@perdasilva: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ad7c5930-6c70-11f0-881b-c69e0b18b9cb-0

@tmshort
Copy link
Contributor

tmshort commented Jul 29, 2025

/retest-required

@tmshort
Copy link
Contributor

tmshort commented Jul 29, 2025

/test e2e-gcp-ovn-builds

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 29, 2025

@perdasilva: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback 751a111 link false /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
ci/prow/e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview 751a111 link false /test e2e-metal-ipi-ovn-bgp-virt-dualstack-techpreview
ci/prow/e2e-aws-ovn-single-node-techpreview 751a111 link false /test e2e-aws-ovn-single-node-techpreview
ci/prow/e2e-aws-ovn-serial-publicnet-1of2 c9e6602 link false /test e2e-aws-ovn-serial-publicnet-1of2
ci/prow/e2e-aws-ovn-serial-publicnet-2of2 c9e6602 link false /test e2e-aws-ovn-serial-publicnet-2of2
ci/prow/okd-e2e-gcp f8a8aa0 link false /test okd-e2e-gcp
ci/prow/e2e-azure-ovn-etcd-scaling cce8714 link false /test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-gcp-fips-serial-1of2 cce8714 link false /test e2e-gcp-fips-serial-1of2
ci/prow/e2e-aws-disruptive cce8714 link false /test e2e-aws-disruptive
ci/prow/e2e-gcp-ovn-techpreview-serial-1of2 cce8714 link false /test e2e-gcp-ovn-techpreview-serial-1of2
ci/prow/e2e-gcp-ovn-techpreview-serial-2of2 cce8714 link false /test e2e-gcp-ovn-techpreview-serial-2of2
ci/prow/e2e-openstack-ovn cce8714 link false /test e2e-openstack-ovn
ci/prow/e2e-aws-ovn-single-node-upgrade cce8714 link false /test e2e-aws-ovn-single-node-upgrade
ci/prow/e2e-gcp-fips-serial-2of2 cce8714 link false /test e2e-gcp-fips-serial-2of2
ci/prow/e2e-aws-ovn-etcd-scaling cce8714 link false /test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-gcp-ovn-etcd-scaling cce8714 link false /test e2e-gcp-ovn-etcd-scaling
ci/prow/e2e-azure-ovn-upgrade cce8714 link false /test e2e-azure-ovn-upgrade
ci/prow/e2e-aws-ovn-single-node cce8714 link false /test e2e-aws-ovn-single-node
ci/prow/e2e-aws-ovn-cgroupsv2 cce8714 link false /test e2e-aws-ovn-cgroupsv2
ci/prow/e2e-metal-ipi-virtualmedia cce8714 link false /test e2e-metal-ipi-virtualmedia
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6 cce8714 link false /test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/e2e-vsphere-ovn-etcd-scaling cce8714 link false /test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-gcp-disruptive cce8714 link false /test e2e-gcp-disruptive
ci/prow/e2e-openstack-serial cce8714 link false /test e2e-openstack-serial

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@tmshort
Copy link
Contributor

tmshort commented Jul 29, 2025

All required tests are passing!

defer g.GinkgoRecover()
oc := exutil.NewCLIWithoutNamespace("openshift-operator-controller-webhooks")

g.BeforeAll(func(ctx g.SpecContext) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You cannot use beforeall like this in origin, it's not true ginkgo. Every test has to be completely independent because they are run in their own process -- so BeforeAll effectively becomes BeforeEach.. If you need a cluster configuration the job would be dedicated to this purpose, configure it the right way and then run your suite.

OTE has designs for tests tied to configs, but it's not implemented yet. https://github.com/openshift/enhancements/blob/master/enhancements/testing/openshift-tests-extension.md

@anik120
Copy link
Contributor

anik120 commented Aug 6, 2025

/close

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 6, 2025
@openshift-merge-robot
Copy link
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci openshift-ci bot closed this Aug 6, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 6, 2025

@anik120: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

e2e-images-update Related to images used by e2e tests jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. vendor-update Touching vendor dir or related files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants