Frontend test cancellations caused by nx parallel: 3 change exposing uncleaned test resources

[spbolton]

Problem Summary

Frontend unit tests in PR workflows are cancelling unexpectedly on the latest commit without any new code pushes.

CRITICAL: This issue is DETERMINISTIC. It fails 100% of the time when frontend tests run (after Dec 23), and 0% of the time when they don't run.

Expected vs Unexpected Cancellations

✅ EXPECTED: Concurrency Control Cancellation

When it happens:

• Developer pushes new commit to PR branch
• Existing workflow run for older commit is automatically cancelled
• New workflow starts for the latest commit

How it appears:

• Job status: "cancelled"
• Reason: Concurrency group policy (cancel-in-progress)
• Message: "This run was cancelled because a newer run was triggered"

❌ UNEXPECTED: Test Hang Cancellation (This Issue)

When it happens:

• No new commits pushed
• Workflow runs on the latest commit
• Tests complete successfully (all pass)
• Process hangs for ~3 minutes with no output
• GitHub Actions detects unresponsiveness
• External shutdown signal sent

How it appears:

• Job status: "cancelled" (not "failed")
• Tests: All pass (e.g., "Tests: 242 passed, 242 total")
• Worker failures: 4-5 occurrences: "A worker process has failed to exit gracefully..."
• ~3 minute silence after tests complete
• Message: "##[error]The runner has received a shutdown signal"
• Message: "##[error]The operation was canceled"

Root Cause Analysis

The Deterministic Pattern

The issue ONLY occurs when:

1. PR changes core-web test files
2. nx affected selects frontend test projects
3. "Frontend Unit Tests" job actually runs

The issue NEVER occurs when:

• PRs don't touch core-web
• PRs touch core-web but not test files
• nx affected skips frontend tests
• "Frontend Unit Tests" job doesn't run

Why Dec 15-22 Had "No Issues"

Dec 15: PR #34099 merged (introducing the configuration change)
Dec 15-22: All PRs either:

• Didn't touch core-web at all
• Changed core-web but not test files
• nx affected found no affected tests
• Frontend Unit Tests job never ran
• Builds appeared successful

Dec 23: First PR that changed core-web test files

• nx affected found affected tests
• Frontend Unit Tests job ran for first time since PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099
• Job cancelled immediately
• Issue became visible

Breaking Change: PR #34099 (December 15, 2025)

Commit: 6771f7ef98
Title: "34098 task migrate angular projects from browser esbuild to the new application builder"
Merged: December 15, 2025 at 22:25 UTC

Critical Change:

# core-web/nx.json line 7
"options": {
-   "parallel": 1
+   "parallel": 3
}

Other Changes in Same PR:

• Build system: webpack → esbuild
• TypeScript: module: es2022, esModuleInterop: true
• HMR: Hot module replacement enabled
• Proxy config: WebSocket changes
• Watch mode: Continuous watch enabled

Note: Analysis shows tests run sequentially despite parallel: 3 setting, but the configuration may still affect Nx's internal process management behavior.

Why PR #34099 Itself Didn't Fail

Critical Discovery: PR #34099 HAD THE EXACT SAME ISSUE but succeeded anyway!

PR #34099 Run 20245075272 (Dec 15, 2025):

19:58 - Frontend tests start
20:08:40 - ⚠️  Worker failure #1: "A worker process has failed to exit gracefully..."
20:09:08 - ⚠️  Worker failure #2
20:10:19 - ⚠️  Worker failure #3
[Multiple worker failures throughout]
20:21:12 - ✅ "Ran all test suites" (LAST TEST OUTPUT)
[3 minutes 32 seconds of silence - NO OUTPUT]
20:24:44 - ✅ "BUILD SUCCESS"
20:25:10 - ✅ Job completes successfully

Result: ✅ SUCCESS - No shutdown signal sent

Failing PRs (Dec 23+):

Tests start
[Multiple worker failures]
XX:XX:XX - ✅ "Ran all test suites" (LAST TEST OUTPUT)
[~3 minutes of silence - NO OUTPUT]
XX:XX:XX - ❌ "##[error]The runner has received a shutdown signal"
XX:XX:XX - ❌ "##[error]The operation was canceled"

Result: ❌ CANCELLED - Shutdown signal sent

Key Difference:

• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099: 3.5 min delay tolerated → Success
• Dec 23+ PRs: 3 min delay NOT tolerated → Cancelled

Most Likely Explanation: GitHub Actions infrastructure changed between Dec 15-23:

• Runner health check timeout may have been reduced
• Health check algorithm changed
• Ubuntu 24.04 runner behavior different
• More aggressive unresponsiveness detection

Underlying Technical Debt

Tests have uncleaned resources (existed BEFORE PR #34099):

• Active timers (setTimeout, setInterval) not cleared
• RxJS subscriptions not unsubscribed
• Unresolved promises
• Zone.js async operations
• Angular change detection intervals
• PrimeNG component internal timers

Result:

• Jest workers fail to exit gracefully (4-5 per run)
• Jest parent process hangs waiting for cleanup
• Process becomes unresponsive (no log output)
• GitHub Actions detects hung process
• External shutdown signal sent (after Dec 23)
• Job cancelled (not failed, because tests passed)

Note: This technical debt existed before PR #34099, but GitHub's tolerance for it appears to have changed.

Evidence

Example Cancelled Run: 20622139223 (Dec 31, 2025)

16:14:54 - ✅ ALL TESTS PASS: "Test Suites: 33 passed, 33 total"
16:14:54 - ⚠️  Jest worker fails to exit gracefully (5th occurrence)
16:14:54 - ✅ "Ran all test suites" - TESTS COMPLETE
[~3 minute hang - Jest waiting for workers to cleanup]
16:17:54 - ❌ "The runner has received a shutdown signal"
16:17:55 - ❌ "The operation was canceled"

Job Presence Analysis

Successful Builds (Dec 15-22):

• Run 20341280316 (Dec 18): ❌ NO "Frontend Unit Tests" job → ✅ Success
• Run 20316649587 (Dec 17): ❌ NO "Frontend Unit Tests" job → ✅ Success

Cancelled Builds (Dec 23+):

• Run 20641205755 (Jan 1): ✅ HAS "Frontend Unit Tests" job → ❌ Cancelled
• Run 20622139223 (Dec 31): ✅ HAS "Frontend Unit Tests" job → ❌ Cancelled

Sequential Execution Observation

Despite parallel: 3 configuration, tests execute sequentially:

16:05:08 - nx run global-store:test ✅
16:05:44 - nx run data-access:test ✅        (36s later)
16:06:57 - nx run edit-content-bridge:test ✅ (73s later)
16:07:14 - nx run edit-ema-ui:test ✅        (17s later)
[Each project starts AFTER previous completes]

This suggests actual parallel execution doesn't happen, but the parallel: 3 configuration may still affect Nx's internal process initialization or resource management.

Affected PRs

Heavily affected (all change core-web test files):

• 34029 browser component #34126 (34029-browser-component) - Changes many test files
• refactor: replace CoreWebService with HttpClient in various services #34165 (issue-34166-usage-ui) - Changes test files
• Update usage dashboard UI to match official dotCMS UI #34168, Issue 34149 update dashboard queries #34179 - Change test files

Pattern: Any PR that changes core-web test files triggers nx affected to run frontend tests, which then cancels 100% of the time.

Solutions

Option 1: Revert Parallelism (Immediate, Recommended for Testing)

Change: core-web/nx.json line 7

-   "parallel": 3
+   "parallel": 1

Why test this despite PR #34099 working:

• Quick to validate (2-3 runs, issue is deterministic)
• Low risk, easy to revert if ineffective
• May help even if mechanism unclear
• If it works, unblocks PRs immediately

Validation:

• Create PR that touches any core-web test file
• Apply revert
• Test with 2-3 runs (not 20-30, since issue is deterministic)
• If Frontend Unit Tests succeed → Keep revert
• If still cancels → Try other solutions

Pros:

• Simple one-line change
• Easy to test and rollback
• May restore whatever made PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 succeed

Cons:

• Doesn't explain why PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 worked with parallel: 3
• Doesn't fix underlying test quality
• May not help if issue is GitHub infrastructure

Option 2: Force Jest Exit (Bandaid, Not Recommended)

Change: Add to core-web/jest.preset.js:

module.exports = {
    ...nxPreset,
    forceExit: true,  // Force Jest to exit after tests complete
    // ... existing config
};

Pros:

• Jobs complete successfully
• Keeps parallel: 3

Cons:

• Masks the problem completely
• Open handles remain undetected
• Technical debt accumulates
• May hide real issues
• Not addressing root cause

Option 3: Configurable Parallelism (Long-term Enhancement)

Allow different settings for local dev vs CI:

// core-web/nx.json
{
  "tasksRunnerOptions": {
    "default": {
      "options": {
        "parallel": process.env.CI ? 1 : 3  // 1 for CI, 3 for local
      }
    }
  }
}

Benefits:

• Developers get fast parallel execution locally
• CI gets stable sequential execution
• Can increase CI parallelism after tests are fixed
• Flexible for future adjustments

Proper Fix (REQUIRED Regardless of Other Solutions)

The underlying issue must be fixed even if revert works:

cd core-web
nx affected -t test --base=main --detectOpenHandles

For each identified test file:

1. Add proper cleanup in afterEach() blocks
2. Clear timers: clearTimeout(), clearInterval()
3. Unsubscribe from RxJS: .unsubscribe()
4. Destroy Angular components properly
5. Clean up Zone.js async operations

Example:

describe('MyComponent', () => {
  let timers: NodeJS.Timeout[] = [];
  let subscriptions: Subscription[] = [];

  afterEach(() => {
    timers.forEach(timer => clearTimeout(timer));
    timers = [];
    subscriptions.forEach(sub => sub.unsubscribe());
    subscriptions = [];
  });

  it('should work', () => {
    const timer = setTimeout(() => { ... }, 1000);
    timers.push(timer);

    const sub = observable$.subscribe(...);
    subscriptions.push(sub);
  });
});

Why this is required:

• Fixes actual technical debt
• Prevents issues even if GitHub behavior changes again
• Improves test reliability
• May indicate production issues
• Enables future parallelism if desired

Recommended Action Plan

Immediate (Today)

1. Apply revert: Change parallel: 3 → 1 in core-web/nx.json
2. Test on any PR that touches core-web tests (only need 2-3 runs since deterministic)
3. If successful: Merge revert to unblock PRs immediately
4. If unsuccessful: Try Option 2 (forceExit) as emergency fallback while investigating

Short-Term (This Week)

1. Run nx affected -t test --base=main --detectOpenHandles locally
2. Identify all tests with open handles
3. Create issues for fixing each leaking test file
4. Begin fixing highest-impact tests

Medium-Term (Next Sprint)

1. Fix all identified leaking tests
2. Remove forceExit if it was used
3. Add ESLint rules for common leak patterns:
   • Detect uncleaned timers
   • Require cleanup in afterEach
   • Flag missing unsubscribe
4. Implement pre-commit hook with --detectOpenHandles
5. Document testing best practices

Long-Term (Next Quarter)

1. Consider restoring parallel: 3 ONLY after all tests are clean
2. Implement configurable parallelism (Option 3)
3. Add continuous monitoring for worker failures
4. Add CI checks to prevent regression
5. Monitor GitHub Actions for infrastructure changes

Key Takeaways

1. Issue is DETERMINISTIC: Fails 100% when frontend tests run (after Dec 23), 0% when they don't
2. Not probabilistic: Only need 2-3 test runs to validate fixes, not 20-30
3. Dec 15-22 "success": PRs simply never triggered frontend tests via nx affected
4. Dec 23+ failures: PRs changed test files → triggered frontend tests → revealed issue
5. PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 paradox: Had same worker failures but succeeded - suggests GitHub infrastructure changed
6. Revert worth testing: Quick validation even if we don't understand why PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 worked
7. Must fix root cause: Regardless of revert effectiveness, underlying test quality must be addressed

[spbolton]

Problem Summary

Frontend unit tests in PR workflows are being cancelled unexpectedly on the latest commit without any new code pushes. This issue affects multiple PRs and started manifesting on December 23-26, 2025.

Critical Distinction: This is NOT the normal concurrency-control cancellation that occurs when developers push new commits. These are unexpected cancellations happening on the latest code.

Expected vs Unexpected Cancellations

✅ EXPECTED: Concurrency Control Cancellation

When it happens:

• Developer pushes new commit to PR branch
• Existing workflow run for older commit is automatically cancelled
• New workflow starts for the latest commit

How it appears in logs:

• Job status: "cancelled"
• Reason: Concurrency group policy (cancel-in-progress)
• Message: "This run was cancelled because a newer run was triggered"
• Expected behavior: Older runs should cancel, latest run should complete

❌ UNEXPECTED: Test Hang Cancellation (Current Issue)

When it happens:

• No new commits pushed
• Workflow runs on the latest commit
• Tests complete successfully (all pass)
• Process hangs for ~3 minutes with no output
• GitHub Actions detects unresponsiveness
• External shutdown signal sent

How it appears in logs:

• Job status: "cancelled" (not "failed")
• Tests: All pass (e.g., "Tests: 242 passed, 242 total")
• Worker failures: 4-5 occurrences per run: "A worker process has failed to exit gracefully..."
• ~3 minute silence after tests complete
• Message: "##[error]The runner has received a shutdown signal"
• Message: "##[error]The operation was canceled"
• Unexpected behavior: Latest run cancels without new commits

Root Cause Analysis

Breaking Change: PR #34099 (December 15, 2025)

Commit: 6771f7ef98
Title: "34098 task migrate angular projects from browser esbuild to the new application builder"
Merged: December 15, 2025 at 22:25 UTC

The Critical Change:

# core-web/nx.json line 7
"options": {
-   "parallel": 1
+   "parallel": 3
}

Why This Exposed the Issue

Before (parallel: 1):

• Tests ran sequentially, one project at a time
• Worker cleanup issues isolated to single projects
• Lower resource usage (~4 Node.js processes)
• Issue masked by sequential execution

After (parallel: 3):

• UP TO 3 test projects can run simultaneously
• Higher resource usage (up to 12 processes: 3 projects × 3 workers + 3 parents)
• Worker failures accumulate faster
• Existing technical debt (uncleaned test resources) becomes visible
• Process more likely to become unresponsive
• GitHub Actions health checks trigger after ~3 minutes

Underlying Technical Debt

Tests have uncleaned resources:

• Active timers (setTimeout, setInterval) not cleared
• RxJS subscriptions not unsubscribed
• Unresolved promises
• Zone.js async operations
• Angular change detection intervals
• PrimeNG component internal timers

Result:

• Jest workers fail to exit gracefully
• Jest parent process hangs waiting for cleanup
• Process becomes unresponsive (no log output)
• GitHub Actions detects hung process
• External shutdown after ~3 minutes

Timeline

Date	Event
Dec 15, 2025 22:25	PR #34099 merged (parallel: 1 → 3)
Dec 23, 2025	First cancellations appear
Dec 26, 2025	Spike: 10+ cancelled runs in one day
Dec 31, 2025 - Jan 1, 2026	Pattern continues across multiple PRs

Delay: 8-11 days between merge and manifestation (PRs created before Dec 15 were in flight)

Affected PRs

Sample of affected PRs (all showing same pattern):

• 34029 browser component #34126 (34029-browser-component)
• refactor: replace CoreWebService with HttpClient in various services #34165 (issue-34166-usage-ui)
• Update usage dashboard UI to match official dotCMS UI #34168
• Issue 34149 update dashboard queries #34179

Pattern: All PRs merging with main after December 15, 2025 are potentially affected.

Evidence

Example Run: 20622139223 (Dec 31, 2025)

16:14:54 - ✅ ALL TESTS PASS: "Test Suites: 33 passed, 33 total"
16:14:54 - ⚠️  Jest worker fails to exit gracefully (5th occurrence)
16:14:54 - ✅ "Ran all test suites" - TESTS COMPLETE
[~3 minute hang - Jest waiting for workers to cleanup]
16:17:54 - ❌ "The runner has received a shutdown signal"
16:17:55 - ❌ "The operation was canceled"

Consistent Pattern Across All Affected Runs:

1. All tests pass (242/242 or similar)
2. 4-5 worker exit failures during execution
3. ~3 minute silence after completion
4. External shutdown signal
5. Job marked as "cancelled"

Solutions

Option 1: Revert Parallelism (Immediate, Safest)

Change: core-web/nx.json line 7

-   "parallel": 3
+   "parallel": 1

Pros:

• Immediate resolution
• Restores working state
• Allows PRs to merge

Cons:

• Loses performance benefits
• Doesn't fix underlying issue
• Slower test execution

Option 2: Force Jest Exit (Immediate, Bandaid)

Change: Add to core-web/jest.preset.js:

module.exports = {
    ...nxPreset,
    forceExit: true,  // Force Jest to exit after tests complete
    // ... existing config
};

Pros:

• Jobs complete successfully
• Keeps parallel: 3 benefits
• Quick fix

Cons:

• Masks the problem
• Open handles remain undetected
• Technical debt accumulates

Option 3: Make Parallelism Configurable (Recommended Long-Term)

Allow different settings for local development vs CI/CD:

Approach A: Environment Variable

// core-web/nx.json
{
  "tasksRunnerOptions": {
    "default": {
      "options": {
        "parallel": process.env.CI ? 1 : 3  // 1 for CI, 3 for local dev
      }
    }
  }
}

Approach B: CI-Specific Configuration

# .github/test-matrix.yml
frontend:
  suites:
    - name: "Frontend Unit Tests"
      maven_args: "test -pl :dotcms-core-web"
      nx_parallel: 1  # Override for CI

Benefits:

• Developers get fast parallel execution locally (parallel: 3)
• CI gets stable sequential execution (parallel: 1)
• Temporary solution while fixing tests
• Can increase CI parallelism once tests are fixed

Proper Fix (Mandatory, Regardless of Option Chosen)

Identify and fix leaking tests:

cd core-web
nx affected -t test --base=main --detectOpenHandles

For each identified test file:

1. Add proper cleanup in afterEach() blocks
2. Clear timers: clearTimeout(), clearInterval()
3. Unsubscribe from RxJS: .unsubscribe()
4. Destroy Angular components properly
5. Clean up Zone.js async operations

Example Fix:

describe('MyComponent', () => {
  let timers: NodeJS.Timeout[] = [];
  let subscriptions: Subscription[] = [];

  afterEach(() => {
    // Clear all timers
    timers.forEach(timer => clearTimeout(timer));
    timers = [];

    // Unsubscribe all subscriptions
    subscriptions.forEach(sub => sub.unsubscribe());
    subscriptions = [];
  });

  it('should work', () => {
    const timer = setTimeout(() => { ... }, 1000);
    timers.push(timer);

    const sub = observable$.subscribe(...);
    subscriptions.push(sub);
  });
});

Recommended Action Plan

Immediate (Today)

1. Choose temporary fix:

   • Option 1 (revert to parallel: 1) - Safest, recommended
   • Option 2 (add forceExit) - Faster but masks issue
   • Option 3 (configurable parallelism) - Best of both worlds

2. Deploy fix to unblock PRs

Short-Term (This Week)

1. Run nx affected -t test --detectOpenHandles locally
2. Identify top 5-10 leaking test files
3. Create issues for fixing each file
4. Begin fixing tests

Medium-Term (Next Sprint)

1. Fix all identified leaking tests
2. Remove forceExit if used
3. Consider restoring parallel: 3 (or make configurable)
4. Add ESLint rules for common patterns
5. Implement pre-commit hook with --detectOpenHandles

Long-Term (Next Quarter)

1. Continuous monitoring for new leaks
2. Update testing documentation
3. Add CI checks to prevent regression
4. Create test utilities for auto-cleanup

Prevention

1. Pre-commit Hook:

   nx affected -t test --detectOpenHandles --uncommitted

2. ESLint Rules:

   • Detect uncleaned timers
   • Require cleanup in afterEach
   • Flag missing unsubscribe

3. CI Validation:

   • Periodic full runs with --detectOpenHandles
   • Resource usage monitoring
   • Alert on worker failure patterns

4. Documentation:

   • Update testing best practices
   • Document common patterns and pitfalls
   • Create cleanup utility functions

Additional Context

Analysis Documents: Comprehensive analysis available in .claude/diagnostics/:

• breaking-change-identification.md - Full timeline and evidence
• frontend-test-cancellation-root-cause.md - Technical details
• runner-shutdown-trigger-analysis.md - GitHub Actions shutdown mechanism
• resource-exhaustion-analysis.md - Confirms NO resource exhaustion

Related PRs:

• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 - Build system migration (triggered issue)
• PR 34087 task security upgrade angular to v20.31.5  #34088 - Angular upgrade to 19.2.9 (merged Dec 12)

GitHub Actions Runs:

• Run 20622139223 - Example showing full pattern
• Run 20641205755 - Another example with same behavior
• Run 20524582180 - Dec 26 cancellation

[spbolton]

Critical Finding: Tests Run Sequentially (But parallel: 3 May Still Affect Execution)

Evidence from Logs

Analysis of run 20641205755 shows tests execute sequentially, not in parallel:

16:05:08 - nx run global-store:test ✅
16:05:44 - nx run data-access:test ✅        (36s later)
16:06:57 - nx run edit-content-bridge:test ✅ (73s later)
16:07:14 - nx run edit-ema-ui:test ✅        (17s later)
16:07:46 - nx run portlets-dot-experiments-data-access:test ✅
16:08:19 - nx run ui:test ✅
16:09:13 - nx run portlets-dot-analytics-data-access:test ✅
16:09:26 - nx run content-drive-ui:test ✅
16:12:07 - nx run template-builder:test ✅
16:13:30 - nx run portlets-dot-experiments-portlet:test ✅

Each project starts AFTER the previous completes. If running in parallel, we'd see 3 projects starting simultaneously.

Why This Matters

Resource exhaustion is NOT the issue:

• Only 1 test project at a time
• ~4 Node.js processes (1 Jest parent + 3 workers)
• Memory: ~800MB-1GB (not 3-4GB)
• Well within runner capacity (16GB RAM, 4 cores)

Why parallel: 3 Could STILL Be the Trigger

Even though actual parallel execution doesn't happen, the parallel: 3 configuration could still affect:

1. Nx internal process spawning behavior
2. Worker pool initialization (reserved but not used)
3. Memory allocation patterns
4. Node.js process management
5. Resource reservation/planning by Nx

The setting might influence how Nx sets up its execution environment, even without actual parallelism.

Quick Validation: Revert and Test

Fastest way to confirm: Empirical test

# core-web/nx.json line 7
-   "parallel": 3
+   "parallel": 1

Expected Results:

• ✅ Tests complete successfully → parallel: 3 WAS the trigger (mechanism unclear but confirmed)
• ❌ Tests still cancel → parallel: 3 NOT involved, investigate other PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes

Benefits of this approach:

1. Fast validation - Single line change, immediate feedback
2. Safe change - Can be kept or reverted based on results
3. Data-driven - Empirical evidence vs speculation
4. Low risk - Already running sequentially anyway

Alternative: Other PR #34099 Changes May Be the Cause

PR #34099 changed more than just parallel:

• Build system: webpack → esbuild
• TypeScript: module: es2022, esModuleInterop: true
• HMR: Hot module replacement enabled
• Proxy config: WebSocket changes
• Watch mode: Continuous watch enabled

If reverting parallel: 3 → 1 doesn't fix the issue, one of these other changes likely altered test cleanup behavior.

Updated Root Cause Assessment

Primary Cause (Confirmed):

• Tests have uncleaned resources (timers, subscriptions, promises)
• Jest workers fail to exit gracefully (4-5 per run)
• Process hangs → GitHub Actions detects unresponsiveness → shutdown

Triggering Change (To Validate):

• parallel: 3 configuration (even without actual parallel execution)
• OR other PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes (TypeScript, build system, etc.)

Recommended Immediate Action

Step 1: Revert parallel to 1

cd core-web
# Edit nx.json line 7: parallel: 3 → 1
git commit -m "fix(ci): Revert nx parallel to 1 to resolve test cancellations"
git push

Step 2: Monitor next 3-5 PR runs

• If fixed: Keep the revert, issue resolved
• If not fixed: Revert the revert, investigate other PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes

Step 3: Fix underlying test quality issues (regardless)

nx affected -t test --base=main --detectOpenHandles

Fix cleanup in identified tests to prevent future issues.

Resource Concern for Future

If we enable actual parallel execution later:

1. Test with parallel: 2 first before increasing
2. Monitor runner memory usage during execution
3. Consider environment-based configuration:
   • parallel: 3 for local dev (developer machines)
   • parallel: 1 for CI/CD (limited runner resources)
4. Profile memory per project to determine safe maximum

Next Steps

1. ✅ Immediate: Revert parallel: 3 → 1 and observe results
2. 🔍 If revert doesn't work: Investigate other PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes (TypeScript, esbuild, etc.)
3. 🛠️ Regardless of cause: Fix leaking tests with --detectOpenHandles
4. 📊 Long-term: Add monitoring and prevention strategies

[spbolton]

CRITICAL FINDING: Issue is DETERMINISTIC - Fails Every Time Frontend Tests Run

Breakthrough Discovery

Your hypothesis was 100% correct! The issue is NOT probabilistic - it's completely deterministic.

The Pattern

SUCCESS: Builds that don't touch core-web

• nx affected finds no affected frontend tests
• Frontend Unit Tests job doesn't even run
• Build succeeds

FAILURE: Builds that touch core-web

• nx affected finds affected frontend tests
• Frontend Unit Tests job runs
• Job cancels EVERY TIME

Evidence

Successful Builds (Dec 15-22): NO Frontend Tests Job

Run 20341280316 (Dec 18, SUCCESS - issue-33902-images-on-block-editor-not-rendering):

Jobs:
- Initialize / Initialize
- PR Build / Initial Artifact Build
- PR Test / Postman Tests
- PR Test / Integration Tests
- PR Test / JVM Unit Tests
- PR Test / CLI Tests
- Finalize / Final Status

❌ NO "Frontend Unit Tests" job → Success

Run 20316649587 (Dec 17, SUCCESS - same branch):

❌ **NO "Frontend Unit Tests" job** → Success

Failed Builds (Dec 23+): HAS Frontend Tests Job

Run 20641205755 (Jan 1, CANCELLED - 34029-browser-component):

Jobs:
- Initialize / Initialize
- PR Build / Initial Artifact Build
✅ PR Test / Frontend Unit Tests  ← PRESENT
- PR Test / Postman Tests
- PR Test / Integration Tests

✅ HAS "Frontend Unit Tests" job → CANCELLED

Run 20622139223 (Dec 31, CANCELLED - issue-34166-usage-ui):

✅ PR Test / Frontend Unit Tests  ← PRESENT

✅ HAS "Frontend Unit Tests" job → CANCELLED

File Change Analysis

PR #34126 (CANCELLED - 34029-browser-component)

Core-web changes: ✅ YES - Many files including:

• core-web/libs/data-access/src/lib/dot-contentlet/dot-contentlet-service.spec.ts
• core-web/libs/edit-content-bridge/src/lib/bridges/angular-form-bridge.spec.ts
• core-web/libs/edit-content/src/lib/fields/**/*.spec.ts
• Multiple .spec.ts test files

nx affected result: Selects multiple test projects → Runs frontend tests → CANCELS

PR #34121 (SUCCESS - issue-33902-images-on-block-editor-not-rendering)

Core-web changes: ✅ YES - But only:

• Component HTML/TS files (no .spec.ts)
• SCSS files
• Store files

nx affected result: No test dependencies changed → Skips frontend tests → SUCCESS

The Complete Picture

PR #34099 merged Dec 15, 2025
    ↓
Dec 15-22: PRs with NO core-web test changes
    → nx affected skips frontend tests
    → Jobs succeed
    → Appears like "no problem"

Dec 23+: PRs with core-web test changes
    → nx affected runs frontend tests
    → Tests pass but workers hang
    → GitHub detects unresponsiveness
    → Job cancelled
    → Issue becomes visible

Why This Changes Everything

1. Issue is NOT Probabilistic

Previously thought:

• Issue has ~30-40% failure rate
• Some builds lucky, some unlucky
• Need 20-30 runs to validate fix

Actually:

• Issue has 100% failure rate when frontend tests run
• 0% failure rate when frontend tests don't run
• Need only 3-5 runs WITH frontend tests to validate

2. Validation Strategy is Much Simpler

OLD approach (assuming probabilistic):

# Apply fix
# Run 20-30 builds
# Calculate statistics
# Hope for significance

NEW approach (knowing it's deterministic):

# Apply fix (revert parallel: 3 → 1)
# Run 3-5 builds that TOUCH core-web tests
# If all 3-5 succeed → Fix is effective
# If any cancel → Fix didn't work

3. Can Immediately Test the Fix

To validate revert:

1. Create test branch that changes ANY core-web test file
2. Apply revert (parallel: 3 → 1)
3. Push and trigger build
4. If succeeds → Revert works
5. If cancels → Need different fix

Example test:

# Create test branch
git checkout -b test-parallel-revert

# Make trivial change to trigger frontend tests
echo "// test comment" >> core-web/libs/data-access/src/lib/dot-contentlet/dot-contentlet-service.spec.ts

# Apply revert
cd core-web
# Edit nx.json: parallel: 3 → 1

# Commit and push
git add .
git commit -m "test: Validate parallel revert fixes cancellations"
git push -u origin test-parallel-revert

# Create PR and watch
gh pr create --title "Test: Validate parallel revert" --body "Testing if parallel: 1 fixes frontend test cancellations"

Updated Root Cause

The issue is triggered by:

1. PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes (parallel: 3 or other changes)
2. PLUS nx affected selecting frontend test projects
3. Tests have uncleaned resources → workers hang
4. GitHub Actions detects unresponsiveness → cancels

Why we didn't see failures Dec 15-22:

• No PRs changed core-web tests
• nx affected never ran frontend tests
• Issue was dormant, waiting for frontend test trigger

Affected Branches Correlation

Heavily affected branches:

• 34029-browser-component → Changes MANY core-web test files
• 33073-migrate-core-web → Migrates core-web (obviously touches tests)
• issue-34146-versioned-indices → Likely touches data-access
• issue-34174-defect-uve-broken-content-navigation → UVE changes

Never affected branches (Dec 15-22):

• Backend-only changes
• Documentation changes
• Frontend changes without test modifications
• Example app changes (NextJS/Astro - different workspace)

Immediate Action

Test the Revert RIGHT NOW

# On any branch that touches core-web tests
cd core-web
# Edit nx.json line 7: parallel: 3 → 1
git add nx.json
git commit -m "fix(ci): Revert nx parallel to 1 to resolve test cancellations"
git push

Expected result:

• If revert works: Frontend tests complete successfully
• If revert doesn't work: Need to investigate other PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 changes

Validation requires only 2-3 runs (not 20-30) because issue is deterministic.

Long-term Solution

Regardless of whether revert works:

1. Fix the underlying issue:

   nx affected -t test --base=main --detectOpenHandles

2. Fix each leaking test

   • Clear timers in afterEach
   • Unsubscribe from observables
   • Destroy components properly

3. Prevent regression:

   • Pre-commit hook with --detectOpenHandles
   • ESLint rules for common patterns
   • CI monitoring for worker failures

4. Consider re-enabling parallel: 3 ONLY after tests are clean

Summary

The "8-day mystery" is solved:

• ❌ NOT a probabilistic failure
• ❌ NOT about runner resources
• ❌ NOT about actual parallel execution
• ✅ DETERMINISTIC: Fails EVERY time frontend tests run
• ✅ Dec 15-22 PRs simply didn't trigger frontend tests
• ✅ Dec 23+ PRs touched core-web → triggered tests → revealed issue

The issue was always there, waiting for frontend tests to run.

[spbolton]

Why PR #34099 Didn't Fail Despite Having the Same Issue

Critical Finding

PR #34099 HAD THE SAME WORKER FAILURES but completed successfully anyway!

Evidence from PR #34099 Final Run (20245075272, Dec 15, 2025)

Worker Failures: YES - Multiple occurrences

20:08:40 - A worker process has failed to exit gracefully...
20:09:08 - A worker process has failed to exit gracefully...
20:10:19 - A worker process has failed to exit gracefully...
[Additional worker failures throughout execution]

Test Completion:

20:21:12 - [INFO] Ran all test suites (LAST TEST OUTPUT)

Build Completion:

20:24:44 - [INFO] BUILD SUCCESS (3 minutes 32 seconds later)

Job Status: ✅ SUCCESS (not cancelled!)

The Difference: Why PR #34099 Succeeded

PR #34099 (Dec 15):

• Tests complete: 20:21:12
• ~3.5 minute delay
• BUILD SUCCESS: 20:24:44
• Job completes normally
• No shutdown signal

Failing PRs (Dec 23+):

• Tests complete: 16:14:54
• ~3 minute delay
• Shutdown signal: 16:17:54
• Job cancelled
• GitHub sends shutdown signal

Possible Explanations

1. GitHub Actions Infrastructure Change (Most Likely)

Hypothesis: GitHub Actions runner health check behavior changed between Dec 15 and Dec 23.

Evidence:

• Dec 15: 3.5 minute silence tolerated → job completed
• Dec 23+: 3 minute silence not tolerated → shutdown signal sent

Possible changes:

• GitHub updated runner software
• Health check timeout reduced (was ~4-5 min, now ~3 min)
• Runner infrastructure upgraded (ubuntu-24.04 rollout)
• Health check algorithm changed

2. Probabilistic Shutdown Timing

Hypothesis: GitHub's health check has some variability/jitter.

Evidence:

• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099: Got lucky, health check passed just before timeout
• Dec 23+ PRs: Unlucky, health check failed before process resumed

Variation observed:

• Some runs: 2m 42s delay before shutdown
• Some runs: 3m 0s delay before shutdown
• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099: 3m 32s delay, NO shutdown

3. Merge Queue vs PR Check Difference

Hypothesis: Merge queue has different health check settings than PR checks.

To investigate:

• Check if PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 was tested via merge queue
• Compare health check behavior in different workflow types

4. Runner Resource Availability

Hypothesis: Dec 15 runners had more resources/better performance.

Evidence:

• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099: 24:51 min total (slower = more time for processes to complete)
• Dec 23+ runs: ~21-23 min total (faster = tighter timing)
• Different runner performance characteristics

Why This Matters

1. Issue Was Present in PR #34099

The underlying problem (uncleaned test resources, worker exit failures) was already present when PR #34099 was merged. It just didn't manifest as a failure.

2. Issue is STILL Deterministic

Even though PR #34099 succeeded, the issue is still deterministic:

• 100% failure rate when frontend tests run (after Dec 23)
• 0% failure rate when frontend tests don't run

The change is in GitHub Actions behavior, not in our code's determinism.

3. Reverting parallel: 3 → 1 May or May Not Help

If the issue is GitHub infrastructure:

• Revert might not help
• Need to fix the actual worker exit failures

If the issue is parallel: 3 configuration:

• Revert should help
• But doesn't explain why PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 worked

4. Must Still Fix Underlying Issue

Regardless of whether revert works, must fix leaking tests:

nx affected -t test --base=main --detectOpenHandles

Even if we get lucky with timing, the technical debt remains.

Recommended Investigation

Step 1: Check GitHub Actions Changes

Look for GitHub announcements around Dec 15-23:

• Runner software updates
• Ubuntu 24.04 runner changes
• Health check algorithm changes

# Compare runner versions
gh run view 20245075272 --log | grep "runner version"
gh run view 20622139223 --log | grep "runner version"

Step 2: Test Revert Empirically

Despite uncertainty about cause, still test the revert:

# Apply revert
cd core-web
# Edit nx.json: parallel: 3 → 1
git commit -m "fix(ci): Revert nx parallel to 1"
git push

# Test on branch that touches core-web tests
# If succeeds: Keep revert (even if we don't understand why PR #34099 worked)
# If fails: Need different solution

Step 3: Add Monitoring

Even if revert works, add monitoring:

# In test script, log timing:
echo "Tests completed at: $(date)"
echo "Build completed at: $(date)"
# Track time between "Ran all test suites" and "BUILD SUCCESS"

Updated Timeline

Dec 15, 2025 - 19:58 to 20:25 (PR #34099):

• Frontend tests run successfully
• Worker failures occur (same as failing runs)
• 3.5 minute delay after test completion
• Build completes successfully (no shutdown)
• PR merges to main

Dec 15-22, 2025:

• PRs don't change core-web tests
• Frontend test job doesn't run
• No issues visible

Dec 23, 2025 - First failures:

• PRs change core-web tests
• Frontend test job runs
• Worker failures occur (same as PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099)
• 3 minute delay after test completion
• Shutdown signal sent (different from PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099!)
• Job cancelled

Dec 23+ - Pattern continues:

• Every PR with frontend tests cancels
• Consistent ~3 minute timeout
• GitHub infrastructure may have changed

Conclusion

The mystery deepens:

• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 had the same underlying issue (worker failures)
• PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 succeeded despite 3.5 min delay
• Dec 23+ PRs cancelled with 3 min delay
• Something changed in GitHub Actions behavior between Dec 15 and Dec 23

Best approach:

1. Test the revert empirically (may work even if we don't understand why PR 34098 task migrate angular projects from browser esbuild to the new application builder #34099 worked)
2. Fix underlying test quality issues (required regardless)
3. Monitor for GitHub Actions infrastructure changes
4. Document the mystery for future reference

The issue is still deterministic in our code (always has worker failures when frontend tests run), but GitHub's tolerance for those failures may have changed.

[spbolton]

Related Issue

Created #34195 to track nightly workflow not running frontend tests. If nightly had been running full test matrix, the frontend test issue would have been detected between Dec 15-22, before affecting PRs on Dec 23.