Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NO-ISSUE: Check IPS on static IPs mode #1127

Merged

Conversation

YuviGold
Copy link
Contributor

/cc @tsorya
/cc @carbonin
/cc @osherdp
/cc @eliorerz

/hold
/test e2e-metal-assisted-networking

@openshift-ci openshift-ci bot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 30, 2021
@YuviGold
Copy link
Contributor Author

/test e2e-metal-assisted-networking

@YuviGold YuviGold force-pushed the static-ips-check-ips branch from 2dfedcf to 7b5e52f Compare September 30, 2021 13:47
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 30, 2021
@openshift-ci
Copy link

openshift-ci bot commented Sep 30, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: osherdp, YuviGold

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@carbonin
Copy link
Member

/hold

Can we wait to see if the networking job succeeds?

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 30, 2021
@osherdp
Copy link
Contributor

osherdp commented Sep 30, 2021

/hold

Can we wait to see if the networking job succeeds?

good idea
🤞

@YuviGold
Copy link
Contributor Author

@carbonin A full installation passed for me locally 🥇

@carbonin
Copy link
Member

@eliorerz mentioned he couldn't actually reproduce the issue locally so I'd rather see it pass in prow

@openshift-ci
Copy link

openshift-ci bot commented Sep 30, 2021

@YuviGold: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-assisted-olm 7b5e52f link false /test e2e-metal-assisted-olm
ci/prow/e2e-metal-assisted-networking 7b5e52f link false /test e2e-metal-assisted-networking

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@YuviGold
Copy link
Contributor Author

YuviGold commented Sep 30, 2021

Job failed on timeout but OVN passed the previous spot, it reached waiting for CVO to be available

<testsuites disabled="0" errors="0" failures="1" tests="6" time="5584.898301362991">
<testsuite disabled="0" errors="0" failures="1" name="TestInstall_test_networking" skipped="0" tests="6" time="5584.898301362991" timestamp="2021-09-30 14:20:43.490569">
<testcase class="fixture" classname="BaseTest" name="cluster[is_static_ip=True, network_type=OVNKubernetes]" time="0.173479"/>
<testcase class="function" classname="Cluster.prepare_for_installation" name="generate_and_download_image[is_static_ip=True, network_type=OVNKubernetes]" time="78.304883"/>
<testcase class="function" classname="Cluster.prepare_for_installation" name="wait_until_hosts_are_discovered[is_static_ip=True, network_type=OVNKubernetes]" time="35.069768"/>
<testcase class="function" classname="Cluster" name="prepare_for_installation[is_static_ip=True, network_type=OVNKubernetes]" time="283.360996"/>
<testcase class="function" classname="Cluster.start_install_and_wait_for_installed" name="start_install[is_static_ip=True, network_type=OVNKubernetes]" time="0.656902"/>
<testcase class="function" classname="Cluster" name="start_install_and_wait_for_installed[is_static_ip=True, network_type=OVNKubernetes]" time="5187.332273">
<failure message="Timeout of 3600 seconds expired waiting for Monitored ['builtin'] operators to be in of the statuses ['available']" type="TimeoutExpired">Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/junit_report/_junit_decorator.py", line 51, in _wrapper value = self._execute_wrapped_function(*args, **kwargs) File "/usr/local/lib/python3.6/site-packages/junit_report/_junit_decorator.py", line 93, in _execute_wrapped_function return self._func(*args, **kwargs) File "/home/assisted/discovery-infra/test_infra/helper_classes/cluster.py", line 508, in start_install_and_wait_for_installed self.wait_for_operators_to_finish() File "/home/assisted/discovery-infra/test_infra/helper_classes/cluster.py", line 670, in wait_for_operators_to_finish fall_on_error_status=False, File "/home/assisted/discovery-infra/test_infra/utils/operators_utils.py", line 77, in wait_till_all_operators_are_in_status waiting_for=f"Monitored {operator_types} operators to be in of the statuses {statuses}", File "/usr/local/lib/python3.6/site-packages/waiting/__init__.py", line 18, in wait for x in iterwait(result=result, *args, **kwargs): File "/usr/local/lib/python3.6/site-packages/waiting/__init__.py", line 56, in iterwait raise TimeoutExpired(timeout_seconds, waiting_for) waiting.exceptions.TimeoutExpired: Timeout of 3600 seconds expired waiting for Monitored ['builtin'] operators to be in of the statuses ['available'] </failure>
</testcase>
</testsuite>
</testsuites>
2021-09-30 15:46:17,902 - root - INFO - 139912761805696 - Asked operators to be in one of the statuses from ['available'] and currently operators statuses are [('console', None, None), ('cvo', 'progressing', 'Unable to apply 4.8.10: some cluster operators have not yet rolled out')]

kube-api operator

+0000 UTC AsExpected Monitoring the cluster} {Disabled False 2021-09-30 14:52:26 +0000 UTC AsExpected }]"
time="2021-09-30T16:12:02Z" level=info msg="Operator kube-apiserver, statuses: [{Degraded True 2021-09-30 14:51:09 +0000 UTC InstallerPodContainerWaiting_ContainerCreating::InstallerPodNetworking_FailedCreatePodSandBox::StaticPods_Error InstallerPodContainerWaitingDegraded: Pod \"installer-9-test-infra-cluster-b705824e-master-0\" on node \"test-infra-cluster-b705824e-master-0\" container \"installer\" is waiting since 2021-09-30 14:54:01 +0000 UTC because ContainerCreating\nInstallerPodNetworkingDegraded: Pod \"installer-9-test-infra-cluster-b705824e-master-0\" on node \"test-infra-cluster-b705824e-master-0\" observed degraded networking: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_installer-9-test-infra-cluster-b705824e-master-0_openshift-kube-apiserver_8db1a928-86ce-4e08-9efc-40d0d5a021ca_0(1cffc7ecde8833c0047261adab3e4fe750594337d69b7ff8ac01fa117c6cbf38): error adding pod openshift-kube-apiserver_installer-9-test-infra-cluster-b705824e-master-0 to CNI network \"multus-cni-network\": [openshift-kube-apiserver/installer-9-test-infra-cluster-b705824e-master-0:ovn-kubernetes]: error adding container to network \"ovn-kubernetes\": CNI request failed with status 400: '[openshift-kube-apiserver/installer-9-test-infra-cluster-b705824e-master-0 1cffc7ecde8833c0047261adab3e4fe750594337d69b7ff8ac01fa117c6cbf38] [openshift-kube-apiserver/installer-9-test-infra-cluster-b705824e-master-0 1cffc7ecde8833c0047261adab3e4fe750594337d69b7ff8ac01fa117c6cbf38] failed to get pod annotation: timed out waiting for annotations\nInstallerPodNetworkingDegraded: '\nStaticPodsDegraded: pods \"kube-apiserver-test-infra-cluster-b705824e-master-0\" not found} {Progressing True 2021-09-30 14:42:15 +0000 UTC NodeInstaller NodeInstallerProgressing: 2 nodes are at revision 0; 1 nodes are at revision 4; 0 nodes have achieved new revision 9} {Available True 2021-09-30 14:45:11 +0000 UTC AsExpected StaticPodsAvailable: 1 nodes are active; 2 nodes are at revision 0; 1 nodes are at revision 4; 0 nodes have achieved new revision 9} {Upgradeable True 2021-09-30 14:41:20 +0000 UTC AsExpected All is well}]"

Anyway going to unhold this one but will keep an eye on what happens with this job.
/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 30, 2021
@openshift-merge-robot openshift-merge-robot merged commit d485251 into openshift:master Sep 30, 2021
YuviGold added a commit to YuviGold/assisted-test-infra that referenced this pull request Oct 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants