Skip to content

NO-JIRA: [TNF] add Two Node Fencing exception to accept less than two etcd endpoints #30058

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −0
Open

NO-JIRA: [TNF] add Two Node Fencing exception to accept less than two etcd endpoints #30058

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions pkg/monitortestlibrary/pathologicaleventlibrary/duplicated_event_patterns.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -499,6 +499,9 @@ func NewUniversalPathologicalEventMatchers(kubeConfig *rest.Config, finalInterva
newCrioReloadedTooOftenEventMatcher := newCrioReloadedTooOftenEventMatcher(finalIntervals)
registry.AddPathologicalEventMatcherOrDie(newCrioReloadedTooOftenEventMatcher)

twoNodeEtcdEndpointsMatcher := newTwoNodeEtcdEndpointsConfigMissingEventMatcher(finalIntervals)
registry.AddPathologicalEventMatcherOrDie(twoNodeEtcdEndpointsMatcher)

return registry
}

Expand Down Expand Up @@ -715,6 +718,20 @@ var CertificateRotation = &SimplePathologicalEventMatcher{
messageReasonRegex: regexp.MustCompile(`^(CABundleUpdateRequired|SignerUpdateRequired|TargetUpdateRequired|CertificateUpdated|CertificateRemoved|CertificateUpdateFailed|CSRCreated|CSRApproved|CertificateRotationStarted|ClientCertificateCreated|NoValidCertificateFound)$`),
}

func newTwoNodeEtcdEndpointsConfigMissingEventMatcher(finalIntervals monitorapi.Intervals) EventMatcher {
// Use topology-based detection for DualReplica (two-node fencing) clusters
dualReplicaTopology := v1.DualReplicaTopologyMode
return &SimplePathologicalEventMatcher{
name: "EtcdEndpointsConfigMissingDuringTwoNodeTests",
locatorKeyRegexes: map[monitorapi.LocatorKey]*regexp.Regexp{
monitorapi.LocatorNamespaceKey: regexp.MustCompile(`^openshift-kube-apiserver-operator$`),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this apply to openshift-apiserver and oauth-apiserver too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I only observed events from openshift-kube-apiserver-operator, do you think oauth-apiserver might incurr in the same error?

},
messageReasonRegex: regexp.MustCompile(`^ConfigMissing$`),
messageHumanRegex: regexp.MustCompile(`apiServerArguments\.etcd-servers has less than two live etcd endpoints`),
topology: &dualReplicaTopology,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is single node filtered out too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this comes from APIServer knowing there is more than 1 endpoint during normal lifecycle, and 1 is a degraded state, but not failure. So I'd say no, only DualReplica.

}
}

// IsEventAfterInstallation returns true if the monitorEvent represents an event that happened after installation.
func IsEventAfterInstallation(monitorEvent monitorapi.Interval, kubeClientConfig *rest.Config) (bool, error) {
if kubeClientConfig == nil {
Expand Down