CI Doctor - System Test - Omni - All Tests Failed (Run #4)

[github-actions]

Summary

All 4 System Test (Omni) jobs failed. The ksail cluster create command could not create clusters because the Sidero Omni API returned an error stating the cluster UUID resource does not exist.

Failure Details

• Run: 25982341723
• Commit: 3b73fd65c9b13c6b6ffd2464e0b48d5b59423b93
• Trigger: schedule (weekly)
• Workflow: System Test - Omni

Root Cause Analysis

The Omni provider (pkg/svc/provider/) failed to create clusters because the Sidero Omni API could not resolve the cluster UUID. The error pattern on all 3 init=true jobs:

rpc error: code = Unknown desc = failed to get cluster UUID: failed to get:
resource ClusterUUIDs.omni.sidero.dev(default/<cluster-name>`@undefined`) doesn't exist

The default-noinit job failed more silently (cluster was never created). This points to an infrastructure/external service issue — either:

1. The Omni service account credentials (OMNI_SERVICE_ACCOUNT_KEY) are expired or misconfigured
2. The Sidero Omni SaaS instance is experiencing connectivity issues
3. The machine class or cluster template referenced (ksail) is no longer available in the Omni instance

Failed Jobs and Errors

Job	Error
System Test (Omni) (false, default-noinit)	Cluster creation silently failed
System Test (Omni) (true, default-init)	ClusterUUIDs.omni.sidero.dev(default/st-omni-default-init-25982341723@undefined) doesn't exist
System Test (Omni) (true, fullstack, ...)	ClusterUUIDs.omni.sidero.dev(default/st-omni-fullstack-25982341723@undefined) doesn't exist
System Test (Omni) (true, altstack, ...)	ClusterUUIDs.omni.sidero.dev(default/st-omni-altstack-25982341723@undefined) doesn't exist

Investigation Findings

• The @undefined suffix in the resource name suggests the cluster identity or template reference is resolving to undefined — this could indicate the cluster template or machine class named ksail is missing from the Omni instance
• All init=true jobs fail at the same point: fetching the kubeconfig from Omni post-create
• The cleanup job (🧹 Cleanup Omni Resources) succeeded, which is expected since no clusters were actually created

Recommended Actions

• Verify the Sidero Omni service account key (OMNI_SERVICE_ACCOUNT_KEY secret) is valid and not expired
• Check the Omni instance for the ksail machine class — it may have been removed or renamed
• Verify connectivity to the Omni endpoint from GitHub Actions runners
• Review whether any recent changes to pkg/svc/provider/omni/ or the Omni provisioner could affect cluster creation

Prevention Strategies

• Add a preflight health-check step in the Omni system test workflow that validates connectivity and credentials before running tests
• Monitor Omni credential expiry and rotate proactively

AI Team Self-Improvement

Omni System Tests: When System Test - Omni fails with rpc error: code = Unknown desc = failed to get cluster UUID ... @undefined``, suspect expired OMNI_SERVICE_ACCOUNT_KEY credentials or a missing machine class in the Omni instance. This is an infrastructure issue, not a code bug.

Historical Context

This is the 4th run of the Omni system tests. No prior failures of this type have been observed in the investigation history.

---

For AI agents: If you create a PR that fixes this issue, include Fixes #<this-issue-number> in the PR description to auto-close it on merge.

Generated by CI Doctor · ● 8.4M · ◷

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/ci-doctor.md@c7d030cd6d4607b90d9ac3ffc8b24aff4f251632

• expires on May 24, 2026, 5:41 AM UTC

[devantler]

🤖 This is an automated response from KSail AI Assistant.

Closing per the 7-day stale policy for CI Doctor - investigation issues (opened 2026-05-17, no activity since).

This does not indicate the failure is resolved — the System Test - Omni scheduled workflow is credential/infra-gated and has been persistently red. If it continues to fail with a current signature, a fresh investigation can be opened rather than reviving this stale one.