Replace jest-worker with custom worker pool#90532
Draft
Conversation
nextjs-bot
added
created-by: Turbopack team
sokra
force-pushed
the
sokra/static-worker-count
branch
from
df326a8 to
4fb6cc5
Compare
Collaborator
Failing test suitesCommit: 5475d6b | About building and testing Next.js
Expand output● build-output-prerender › with a next config file › without --debug-prerender › shows only a single prerender error with a mangled stack
Expand output● build-output-prerender › with a next config file › without --debug-prerender › shows only a single prerender error with a mangled stack
Expand output● Cache Components Errors › Build Without --prerender-debug › Dynamic Metadata - Static Route › should error the build if generateMetadata is dynamic when the rest of the route is prerenderable ● Cache Components Errors › Build Without --prerender-debug › Dynamic Metadata - Error Route › should error the build for the correct reason when there is a cache components violation alongside dynamic metadata ● Cache Components Errors › Build Without --prerender-debug › Dynamic Metadata - Static Route With Suspense › should error the build if generateMetadata is dynamic when the rest of the route is prerenderable ● Cache Components Errors › Build Without --prerender-debug › Dynamic Viewport - Static Route › should error the build if generateViewport is dynamic ● Cache Components Errors › Build Without --prerender-debug › Dynamic Viewport - Dynamic Route › should error the build if generateViewport is dynamic even if there are other uses of dynamic on the page ● Cache Components Errors › Build Without --prerender-debug › Dynamic Root › should error the build if cache components happens in the root (outside a Suspense) ● Cache Components Errors › Build Without --prerender-debug › Sync Dynamic Platform › With Fallback - Math.random() › should error the build if Math.random() happens before some component outside a Suspense boundary is complete ● Cache Components Errors › Build Without --prerender-debug › Sync Dynamic Platform › Without Fallback - Math.random() › should error the build if Math.random() happens before some component outside a Suspense boundary is complete ● Cache Components Errors › Build Without --prerender-debug › Sync Dynamic Request › cookies › should error the build with a runtime error ● Cache Components Errors › Build Without --prerender-debug › Sync Dynamic Request › headers › should error the build with a runtime error ● Cache Components Errors › Build Without --prerender-debug › Error Attribution with Sync IO › Guarded RSC with unguarded Client sync IO › should error the build with a reason related to sync IO access ● Cache Components Errors › Build Without --prerender-debug › Error Attribution with Sync IO › Unguarded RSC with guarded Client sync IO › should error the build with a reason related dynamic data ● Cache Components Errors › Build Without --prerender-debug › Error Attribution with Sync IO › unguarded RSC with unguarded Client sync IO › should error the build with a reason related to sync IO access ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › Inside ● Cache Components Errors › Build Without --prerender-debug › With ● Cache Components Errors › Build Without --prerender-debug › With ● Cache Components Errors › Build Without --prerender-debug › With ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Current Time - Date() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Current Time - Date.now() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Current Time - new Date() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Random - Math.random() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Web Crypto - getRandomValue() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Web Crypto - randomUUID() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - generateKeyPairSync() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - generateKeySync() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - generatePrimeSync() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - getRandomValues() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - random-bytes() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - random-fill-sync() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - random-int-between() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - random-int-up-to() › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › Sync IO - Node Crypto - random-uuid › should error the build if sync IO is used in a Server Component while prerendering ● Cache Components Errors › Build Without --prerender-debug › IO accessed in Client Components › should error the build if IO is accessed in a Client Component |
Collaborator
Stats from current PR🔴 1 regression
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles: **400 kB** → **400 kB** ✅ -13 B80 files with content-based hashes (individual files not comparable between builds) Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URL |
Removes the jest-worker dependency and implements a custom worker pool under packages/next/src/lib/worker/ with lazy spawning, dynamic scaling, per-worker concurrency via request ID correlation, and individual worker restart support. Key changes: - New WorkerPool class that spawns workers on-demand as jobs arrive - Custom child process and worker thread handlers with request IDs for concurrent call multiplexing - Worker class refactored to use WorkerPool instead of JestWorker - Dev server updated to use the Worker wrapper instead of direct jest-worker import - Build trace paths updated for new worker child file locations
Add WorkerPool unit tests (worker-pool.test.ts) covering: - Lazy spawning (no workers on construction, spawn on dispatch) - Dynamic scaling (0 to maxWorkers as jobs arrive) - Task queuing and FIFO dequeuing - Concurrency per worker (multiple in-flight calls) - Message handling (OK, CLIENT_ERROR, SETUP_ERROR, CUSTOM) - end() and close() lifecycle - Unexpected worker exit handling - stdout/stderr piping - Fork options (env, execArgv, JEST_WORKER_ID) Extend Worker wrapper tests (worker.test.ts) covering: - Color propagation edge cases (CI, TERM=dumb, stderr-only TTY) - Option forwarding (concurrencyPerWorker, enableWorkerThreads, maxRetries) - Exposed method behavior (underscore filtering, arg sanitization) - end()/close() lifecycle (idempotency, double-call errors) - Env configuration (IS_NEXT_WORKER, isolatedMemory, enableSourceMaps) - Timeout and activity callbacks - Custom message handling
- Remove stale jest-worker ignore pattern from collect-build-traces.ts
- Remove compiled jest-worker directory and type declarations
- Consolidate instanceof checks into WorkerHandle abstraction class
- Remove dead restartWorker public method
- Fix respawnCount reset bug: preserve count across respawns via
existingRespawnCount parameter
- Fix process.on('exit') listener leak: store handler reference,
remove in end()/close()
- Add error event handler on child processes to prevent unhandled
'error' events (ENOMEM, EMFILE)
- Extract shared child protocol logic into worker-child-common.ts
(ChildTransport interface + createMessageHandler factory)
- Fix maxRetries reference bug in WorkerPool constructor (was
referencing non-existent options.maxRetries instead of maxRespawns)
- Update README with architecture changes and option renames
- Add comprehensive tests: end() in-flight rejection, spawn errors,
respawn count preservation, exit handler cleanup
Limits how many workers can be in the "booting" state simultaneously, preventing resource contention when many tasks arrive at once. Workers send a PARENT_MESSAGE_READY after loading their module and running setup(), transitioning from booting to ready. When the booting limit is reached, tasks queue instead of spawning new workers. When a worker becomes ready, queued tasks trigger additional spawns as needed. Default: Math.ceil(maxWorkers / 4)
- Fix constructor defaults: spread options first, then apply fallbacks, so explicit undefined doesn't clobber computed defaults - Clear booting state on SETUP_ERROR so the booting slot is freed - Validate maxBootingWorkers >= 1 to prevent silent pool deadlock - Remove stale 'Call retries were exceeded' catch guard from type-check (was a jest-worker message, never produced by the new pool) - Remove unused errors.json entry 1083
- Fix constructor defaults: spread options first, then apply fallbacks, so explicit undefined doesn't clobber computed defaults - Clear booting state on SETUP_ERROR so the booting slot is freed - Validate maxBootingWorkers >= 1 to prevent silent pool deadlock - Remove stale 'Call retries were exceeded' catch guard from type-check (was a jest-worker message, never produced by the new pool) - Remove unused errors.json entry 1083
Add missing documentation for pool/worker methods, callbacks (onWorkerExit, onCustomMessage), all Worker options (logger, maxRetries mapping, forkOptions.execArgv), shutdown semantics (500ms force-kill timeout, close vs end), setup/teardown contract details (sync or async, setupArgs), and process exit handler cleanup behavior.
Add entries 1083-1085 generated during build: - 1083: maxBootingWorkers must be at least 1 - 1084: Worker exited during shutdown - 1085: Worker pool ended while requests were in-flight
- Reword "crash/crashes" in worker README to avoid alex profanity lint - Add tsec ban-eval-calls exemption for worker-child-common.ts - Add 6 new dist/lib/worker/*.js files to NFT snapshot - Update CPU profiling test to expect build-static-worker profile
sokra
force-pushed
the
sokra/static-worker-count
branch
from
63886a8 to
f3097b5
Compare
Remove maxRespawns from WorkerPool — the pool already spawns new workers on demand when dispatch() is called, so explicit respawning is unnecessary. Instead, implement retry logic in the Worker class: when a dispatch fails with WorkerExitError (worker crash), re-dispatch the same call up to maxRetries times. - Add WorkerExitError class to distinguish crashes from method errors - Remove maxRespawns, respawnCount, and respawn branch from pool - Add dispatchWithRetry() in Worker with onRestart callback support - Remove onWorkerExit → process.exit() handler from Worker
This env var was inherited from jest-worker and serves no purpose in Next.js's own worker pool implementation.
…mments
- WorkerHandle: replace _isThread boolean + type casts with typed nullable
fields (_thread/_process), eliminating all `as` casts
- Extract _canSpawnWorker() helper to deduplicate the spawn-eligibility
check used in dispatch(), _handleExit(), and _spawnForQueuedTasks()
- Simplify _deserializeError() using destructuring and Object.assign
- Extract buildWorkerEnv() from the Worker constructor to isolate
NODE_OPTIONS, debug port, source maps, and color detection logic
- Add comment explaining the +1 debug port offset
- Extract normalizeError() in worker-child-common.ts to deduplicate
error serialization in reportClientError/reportInitializeError
- Add comments explaining why eval('require') is used
- Rename legacy "Farm" error message to "Worker"
sokra
commented
Feb 26, 2026
| | Option | Type | Default | Description | | ||
| |--------|------|---------|-------------| | ||
| | `exposedMethods` | string[] | required | Methods to wire up from worker module (underscore-prefixed names are skipped) | | ||
| | `debuggerPortOffset` | number | required | Debugger port offset (`-1` = not inspectable) | |
Member
Author
There was a problem hiding this comment.
Better use number | undefined, default to undefined and undefined = not inspectable
sokra
commented
Feb 26, 2026
| |--------|------|---------|-------------| | ||
| | `exposedMethods` | string[] | required | Methods to wire up from worker module (underscore-prefixed names are skipped) | | ||
| | `debuggerPortOffset` | number | required | Debugger port offset (`-1` = not inspectable) | | ||
| | `isolatedMemory` | boolean | required | If true, strips `--max-old-space-size` from NODE_OPTIONS | |
sokra
commented
Feb 26, 2026
| | `exposedMethods` | string[] | required | Methods to wire up from worker module (underscore-prefixed names are skipped) | | ||
| | `debuggerPortOffset` | number | required | Debugger port offset (`-1` = not inspectable) | | ||
| | `isolatedMemory` | boolean | required | If true, strips `--max-old-space-size` from NODE_OPTIONS | | ||
| | `numWorkers` | number | 1 | Maps to `maxWorkers` | |
Member
Author
There was a problem hiding this comment.
Default to the number of CPUs - 1, minimum 1
sokra
commented
Feb 26, 2026
Comment on lines
122
to
125
| // eval('require') prevents the bundler from statically tracing this | ||
| // require(). The module path is provided at runtime via the INITIALIZE | ||
| // message, so static analysis would fail or produce incorrect bundles. | ||
| // eslint-disable-next-line no-eval |
Member
Author
There was a problem hiding this comment.
This is not needed as this code isn't bundled.
sokra
commented
Feb 26, 2026
Comment on lines
155
to
157
| // For dev server, it's not necessary to spin up too many workers as long as you are not doing a load test. | ||
| // This helps reusing the memory a lot. | ||
| numWorkers: 1, |
Member
Author
There was a problem hiding this comment.
add concurrency per worker of 4
…'require')
- Add workerName option to Worker class for user-friendly error messages
(e.g. "Next.js build worker exited with code: X and signal: Y")
- Change debuggerPortOffset from required number to optional (default undefined)
- Change isolatedMemory from required boolean to optional (default false)
- Rename numWorkers to maxWorkers, default to os.cpus().length - 1
- Remove eval('require') wrapper in worker-child-common.ts (not bundled)
- Add concurrencyPerWorker: 4 to dev server worker
- Update all callers and tests
- Static generation worker: "Next.js static worker"
- Turbopack build worker: "Next.js turbopack build worker"
- Webpack build worker: "Next.js webpack worker ({compilerName})"
- Build trace worker: "Next.js build trace worker"
- Type check worker: "Next.js type check worker"
- Static paths worker (dev): "Next.js static paths worker"
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Replaces the vendored
jest-workerdependency with a purpose-built worker pool implementation (packages/next/src/lib/worker/). Deletes the compiledjest-workerbundle and all associated type declarations.Why?
jest-workeris maintained for Jest's needs, not Next.js's. It eagerly spawns all workers at construction and doesn't support per-worker concurrency, boot throttling, or fine-grained crash recovery.next buildperformance with many static pages.How?
New files in
packages/next/src/lib/worker/:worker-pool.tsindex.tsWorkerclass (drop-in replacement): timeout/restart, NODE_OPTIONS, color propagation, exposed methodstypes.tsworker-child-common.tsChildTransport+createMessageHandler)worker-process-child.tschild_processentry point (thin wrapper)worker-thread-child.tsworker_threadsentry point (thin wrapper)README.mdKey features of the new pool:
maxWorkersas concurrent tasks increaseconcurrencyPerWorkerallows multiple in-flight calls per workermaxBootingWorkers(default:ceil(maxWorkers/4)) limits concurrent worker startups. Workers send aPARENT_MESSAGE_READYmessage after loading their module and runningsetup(). When the booting limit is reached, tasks queue instead of spawning new workers.maxRespawnscontrols how many times a worker slot is respawned after unexpected exits. In-flight requests are always rejected; queued tasks are dispatched to healthy workers.Other changes:
collect-build-traces.ts: Updated standalone trace paths fromjest-worker/*Childtoworker-*-childtype-check.ts: Removed stale'Call retries were exceeded'catch guard (jest-worker message that the new pool never produces), removed unnecessary try/catch wrappernext-dev-server.ts: Switched import to newWorkerclass, added required options (debuggerPortOffset,isolatedMemory,exposedMethods), removed manual stdout/stderr piping (handled internally now)tsec-exemptions.json: Addedban-eval-callsexemption forworker-child-common.ts(eval('require')prevents static analysis of dynamic module paths in child process entry points)errors.json: Added error codes 1076–1085 for worker pool error messagessrc/compiled/jest-worker/and removed type declarations fromcompiled.d.ts/$$compiled.internal.d.tsUpdated existing tests:
test/production/next-server-nft: Updated NFT snapshot to include 6 newdist/lib/worker/*.jsfilestest/e2e/cpu-profiling: Updated expected CPU profile counts (+1 forbuild-static-workerprofile)New test coverage:
worker-pool.test.ts: 55 tests covering lazy spawning, initialization, dispatch/message handling, task queuing, concurrency, custom messages, end/close, worker exit handling, stdout/stderr, dynamic scaling, fork options, respawning, maxBootingWorkers (throttling, READY signaling, crash recovery, setup error recovery, validation, shutdown)worker.test.ts: 41 tests covering the high-level Worker class (color propagation, option passthrough, exposed methods, end/close lifecycle, env configuration, timeout/activity, exit handler cleanup)