ci: consolidate 8 tiny integration matrix groups into 4 (31→23 jobs)#17175
ci: consolidate 8 tiny integration matrix groups into 4 (31→23 jobs)#17175
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR consolidates the CI integration test matrix from 31 to 23 parallel jobs by merging 8 short-duration test groups into 4 consolidated groups. The consolidation targets groups with test durations under 2 seconds that spend more time on job setup overhead (~2 minutes) than actual testing, saving approximately 16 minutes of runner time per CI run.
Changes:
- Merged CLI MCP Playwright and CLI MCP Gateway into "CLI MCP Connectivity"
- Merged CLI Audit & Inspect and CLI Logs & Firewall into "CLI Audit Logs & Firewall"
- Merged Workflow Safe Outputs, Workflow GitHub & Git, and Workflow Misc Part 1 into "Workflow Features"
- Merged five small workflow groups (Cache, Permissions, Dependabot & Security, Runtime & Setup, String & Sanitization) into "Workflow Infra"
Comments suppressed due to low confidence (1)
.github/workflows/ci.yml:175
- The pattern includes "TestMCPGateway" which does not match any integration tests in pkg/cli. A search of integration test files shows no test functions starting with TestMCPGateway. The only Gateway-related tests (TestGateway*, TestParseGatewayLogs, etc.) are in gateway_logs_test.go which has the build tag "!integration", making them unit tests that won't run in this integration test matrix.
pattern: "TestMCPInspectPlaywright|TestMCPGateway"
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| - name: "CLI Audit Logs & Firewall" | ||
| packages: "./pkg/cli" | ||
| pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow" | ||
| pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|^TestAudit|^TestInspect" |
There was a problem hiding this comment.
The pattern includes "^TestInspect" which does not match any integration tests in pkg/cli. A search of all integration test files in pkg/cli shows no test functions starting with TestInspect. This pattern will not match any tests and should be removed to avoid confusion.
This issue also appears on line 175 of the same file.
| pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|^TestAudit|^TestInspect" | |
| pattern: "TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|^TestAudit" |
Many integration matrix groups had <2s test durations but ~2min setup overhead each (checkout, setup-go, deps, build). Merging the shortest groups cuts 8 jobs per CI run, saving ~16 min of runner time with no wall-clock impact.
Consolidations
The catch-all groups (
CLI Completion & Other,Workflow Misc Part 2) skip_patterns are unchanged — all existing test patterns remain covered.Original prompt
This section details on the original issue you should resolve
<issue_title>[ci-coach] ci: consolidate 8 tiny integration matrix groups into 4</issue_title>
<issue_description>Reduces the integration test matrix from 31 to 23 parallel jobs by merging short-duration groups that spend more time on job setup than actual testing.
Background
Analysis of recent CI artifacts (run §22222481551) showed many integration matrix groups with extremely short actual test durations (<2s), but each requiring ~2 minutes of setup overhead (checkout, setup-go, dependency download, binary build). Consolidating these saves significant runner time without affecting wall clock duration.
Changes
1. CLI MCP Connectivity (saves 1 job)
Merged: CLI MCP Playwright (~1.0s) + CLI MCP Gateway (~0.6s) → CLI MCP Connectivity
New combined pattern:
TestMCPInspectPlaywright|TestMCPGateway2. CLI Audit Logs & Firewall (saves 1 job)
Merged: CLI Audit & Inspect (~0.7s) + CLI Logs & Firewall (~0.7s) → CLI Audit Logs & Firewall
New combined pattern:
TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow|^TestAudit|^TestInspect3. Workflow Features (saves 2 jobs)
Merged: Workflow Safe Outputs (~0.4s) + Workflow GitHub & Git (~1.0s) + Workflow Misc Part 1 (~0.5s) → Workflow Features
New combined pattern:
SafeOutputs|CreatePullRequest|OutputLabel|HasSafeOutputs|GitHub|Git|PushToPullRequest|BuildFromAllowed|TestAgent|TestCopilot|TestCustom|TestEngine|TestModel|TestNetwork|TestOpenAI|TestProvider4. Workflow Infra (saves 4 jobs)
Merged: Workflow Cache (~0.2s) + Workflow Permissions (~0.2s) + Workflow Dependabot & Security (~0.1s) + Workflow Runtime & Setup (~0.2s) + Workflow String & Sanitization (~1.7s) → Workflow Infra
New combined pattern:
^TestCache|TestCacheDependencies|TestCacheKey|TestValidateCache|TestPermissions|TestPackageExtractor|TestCollectPackagesFromWorkflow|Dependabot|Security|PII|Runtime|Setup|Install|Download|Version|Binary|String|Sanitize|Normalize|Trim|Clean|FormatExpected Impact
Safety Analysis
Skip-pattern verification
All merged patterns are already present in the respective catch-all skip_patterns:
CLI Completion & Otherskip_pattern covers:TestMCPInspectPlaywright,TestMCPGateway✅TestLogs|TestFirewall|TestNoStopTime|TestLocalWorkflow✅^TestAudit|^TestInspect✅Workflow Misc Part 2skip_pattern covers:SafeOutputs|CreatePullRequest|OutputLabel|HasSafeOutputs✅GitHub|Git|PushToPullRequest|BuildFromAllowed✅TestAgent|TestCopilot|TestCustom|TestEngine|TestModel|TestNetwork|TestOpenAI|TestProvider✅^TestCache|TestCacheDependencies|TestCacheKey|TestValidateCache✅TestPermissions|TestPackageExtractor|TestCollectPackagesFromWorkflow✅Dependabot|Security|PII✅Runtime|Setup|Install|Download|Version|Binary✅String|Sanitize|Normalize|Trim|Clean|Format✅No tests will be orphaned or duplicated.
Test duration baseline (from CI run §22222481551)
References: §22222481551
🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.