Skip to content

perf(tests): cut per-file import graphs in apps/sim and shard the suite across CI runners - #7367

Merged
waleedlatif1 merged 2 commits into
stagingfrom
perf/speed-up-test-suite
Sep 1, 2026
Merged

perf(tests): cut per-file import graphs in apps/sim and shard the suite across CI runners#7367
waleedlatif1 merged 2 commits into
stagingfrom
perf/speed-up-test-suite

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Profiled the apps/sim suite (2,868 files): wall time is bounded by the single Vite server thread that feeds every worker (flat from 4 to 13 workers), so the cost is module fetches per test file, not CPU. Cumulative import time was 7x the actual test time.
  • vitest.setup.ts now imports the 15 mocks it registers by file instead of the whole @sim/testing barrel (69 modules -> 16 per file); @sim/testing exports ./mocks/*.
  • Global mocks for @/tools/metadata, @/tools/metadata-outputs (5MB generated modules each) and @/components/icons (10k-line brand SVG sheet, reached by 445 files); six tests that assert real metadata opt out with vi.unmock.
  • Billing usage/limit notifications lazy-load the React email templates and mail providers, which every route builder was pulling in through billing attribution (~1.5s/file on 170 route tests). @trigger.dev/core/v3 is mocked in setup for the same chain.
  • Copilot tool-handler registration loads the handler map on first execution instead of at import; the map statically reaches most of lib/ (14-30s of imports on every copilot request test).
  • PostCSS is skipped in tests (css: false); loading the Tailwind config for a .module.css import cost ~150ms/file on 409 files.
  • Nine per-service tool tests that unmocked the full 4k-entry registry (10-45s of imports each) build a partial registry from their own service module via partialToolRegistry, and assert registration through the generated @/tools/tool-ids.
  • Root test runs the 13 script suites in one vitest run (root vitest.config.ts) instead of eleven sequential processes; check:script-test-coverage now asks vitest list which files it collects.
  • CI runs apps/sim as two shards on two runners (SIM_TEST_SHARD, hashed by Turbo); everything else stays in Lint and Test with shard 1.
  • Fixture: mockPineconeResponses no longer calls randomFloat() at module scope (a test that partially mocks @sim/utils/random now imports the factories after its own mocks).

Local full run: 2,861 files, 314s -> 212s wall; cumulative setup 926s -> 385s, import 2,698s -> 1,212s. CI should roughly halve on top of that from the shard.

Type of Change

  • Performance / CI

Testing

Full apps/sim suite green locally (39,397 tests), bun run test:scripts, bun run check:audits, bun run lint:check, bun run type-check in apps/sim and packages/testing.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@waleedlatif1
waleedlatif1 requested a review from a team as a code owner September 1, 2026 19:40
@vercel

vercel Bot commented Sep 1, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Sep 1, 2026 7:56pm UTC

Request Review

@gitguardian

gitguardian Bot commented Sep 1, 2026

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@greptile-apps

greptile-apps Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR reduces test import overhead by introducing targeted global mocks and selective real-module opt-outs, lazily loading expensive billing and copilot modules, consolidating script tests, and splitting the Sim test suite across two CI runners.

  • Adds awaited, memoized lazy registration for copilot tool handlers.
  • Restores real icon and generated-metadata modules in the previously affected identity and projection tests.
  • Replaces full tool-registry imports in service tests with partial registries.
  • Adds two-way CI sharding for the apps/sim test suite.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains.

No blocking failure remains.

Important Files Changed

Filename Overview
apps/sim/lib/copilot/tool-executor/register-handlers.ts Memoizes lazy handler-map loading, while current execution callers await initialization before dispatch.
apps/sim/vitest.setup.ts Introduces lightweight global mocks for expensive modules, with the previously affected tests now opting out where real behavior is required.
apps/sim/lib/copilot/tools/server/blocks/get-blocks-metadata-projection.test.ts Restores both generated metadata modules for the projection coverage identified in the previous review.
apps/sim/blocks/brand-icon.test.tsx Restores real icon exports so component-identity and rendering assertions no longer compare global stubs.
.github/workflows/test-build.yml Runs the Sim suite as two complementary shards while retaining the remaining repository checks in the primary job.

Reviews (2): Last reviewed commit: "fix(tests): recheck abort after lazy han..." | Re-trigger Greptile

Comment thread apps/sim/lib/copilot/tool-executor/handler-map.ts
Comment thread apps/sim/vitest.setup.ts
Comment thread apps/sim/vitest.setup.ts

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against the latest diff

Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Re-trigger cubic

Comment thread apps/sim/lib/copilot/request/tools/executor.ts Outdated
Comment thread scripts/check-script-test-coverage.ts Outdated
Comment thread scripts/check-script-test-coverage.ts Outdated
Comment thread apps/sim/lib/copilot/tool-executor/handler-map.ts
Comment thread apps/sim/vitest.setup.ts
Comment thread apps/sim/lib/copilot/tool-executor/handler-map.ts
@waleedlatif1
waleedlatif1 force-pushed the perf/speed-up-test-suite branch from 2a84e0e to 5c51db6 Compare September 1, 2026 19:52
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant