Nightly workflow not running frontend tests - matrix tests skipped

[spbolton]

Description

During investigation of frontend test cancellations (issue 34194), discovered that nightly workflow is not running frontend tests or other matrix tests. Nightly should run ALL tests to catch issues before they reach PR workflows. This prevented early detection of the worker exit failures that started after PR 34099 merged on Dec 15. Frontend tests would have failed nightly between Dec 15-22, alerting team before PRs were affected on Dec 23. Need to investigate why nightly skips matrix tests and restore full test coverage to nightly runs.

Acceptance Criteria

• [Define what needs to be accomplished]

Additional Notes

[Any additional context or notes]

[spbolton]

Detailed Analysis

Discovery Context

While investigating issue #34194 (frontend test cancellations), discovered that the nightly workflow does not execute matrix tests, including frontend unit tests.

Expected Behavior

Nightly workflow should:

• Run ALL test suites (not just a subset)
• Execute full test matrix (Integration, Postman, Frontend, CLI, etc.)
• Provide comprehensive validation of main branch
• Catch breaking changes before they affect PR workflows

Actual Behavior

Nightly workflow currently:

• Skips matrix tests entirely
• Does not run frontend unit tests
• Misses potential issues in main branch

Impact on Issue #34194

Timeline of what should have happened:

Dec 15, 2025 - PR #34099 merges to main
Dec 15 night - Nightly runs, frontend tests fail/cancel
Dec 16 morning - Team alerted to issue in main
Dec 16 - Issue investigated and fixed

What actually happened:

Dec 15, 2025 - PR #34099 merges to main
Dec 15-22 - Nightly doesn't run frontend tests, issue undetected
Dec 23 - First PR with core-web changes fails
Dec 23+ - Multiple PRs affected, investigation begins
Jan 2 - Root cause identified

Cost of missed detection:

• 8 days of issue being present in main
• Multiple PRs blocked (34126, 34165, 34168, 34179)
• Developer time wasted on "CI failures"
• Investigation effort to identify root cause

What Nightly Should Have Caught

If nightly had been running frontend tests Dec 15-22:

Worker Exit Failures:

• Would have seen 4-5 "worker process failed to exit gracefully" messages
• Would have detected the ~3 minute hang after test completion
• Would have either:
  • Cancelled (same as PR workflows) - alerting team immediately
  • Succeeded (like PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099) - but with visible worker failures in logs

Early Warning Signs:

• Worker failures in nightly logs
• Longer test execution times
• Process unresponsiveness patterns

Investigation Needed

Questions to answer:

1. When did nightly stop running matrix tests?
2. Was this intentional or accidental?
3. What workflow configuration controls this?
4. Are other test suites also missing from nightly?

Files to check:

# Nightly workflow definition
.github/workflows/cicd_nightly.yml

# Workflow includes/reusable workflows
.github/workflows/cicd_comp_*.yml

# Test matrix configuration
.github/test-matrix.yml

# Compare with PR workflow
diff .github/workflows/cicd_1-pr.yml .github/workflows/cicd_nightly.yml

Recommended Fix

Immediate:

1. Identify why nightly skips matrix tests
2. Update nightly workflow to include full test matrix
3. Validate nightly runs all tests (Integration, Frontend, Postman, CLI, Karate, JVM)

Validation:

• Trigger nightly manually
• Verify "Frontend Unit Tests" job appears
• Confirm all matrix test jobs execute

Long-term:

• Monitor nightly for regressions
• Alert if nightly doesn't run expected test jobs
• Document nightly test coverage requirements

Acceptance Criteria

• Nightly workflow configuration updated to include matrix tests
• Frontend Unit Tests job executes in nightly runs
• All test matrix jobs execute in nightly runs (Integration, Postman, CLI, Karate, JVM)
• Manual nightly trigger verified to run all tests
• Scheduled nightly runs confirmed to execute full test suite
• Documentation updated with nightly test coverage expectations

Related Issues

• Frontend test cancellations caused by nx parallel: 3 change exposing uncleaned test resources #34194 - Frontend test cancellations (would have been caught by nightly)

Additional Context

Why this matters:

• Nightly is the safety net for main branch
• Catches issues before they affect developers
• Provides early warning for breaking changes
• Reduces debugging time (issue found closer to introduction)
• Prevents cascading failures across multiple PRs

Current gap:

• PR workflows catch issues (but after code merged)
• No early warning from nightly
• Issues discovered when PRs try to merge
• Reactive instead of proactive detection