[Bug] workflow cache size can grow and crash worker processes with memory error

[millerick]

What are you really trying to do?

Our production system uses a workflow worker to run about 100 distinct workflow_types. We use zod schemas to validate inputs and outputs to workflows and activities. Many of these are defined at the top level of files, all of which appear to be instantiated and have their own allocation in memory for each workflow that is in the cache. We want our production system to be able to survive bursts of workflows starting without the workers crashing.

Describe the bug

The workflow cache size can grow and crash the process with an out of memory error. This can be achieved with the default workflow cache settings. Our production workers use a 6GB memory request/JS heap size and would regularly see crashes due to exhausting memory as the cache size grew. It is particularly noticeable if we received a large burst of new workflows since those would all start around the same time and crashes would happen before the workflows would have a chance to complete--occasionally resulting in all workers crashing and ending up in a crash loop backoff--preventing our system from continuing to process workflows until someone manually scaled up the number of workers to ensure that the average number of workflows per worker was low enough.

These crashes stopped when we explicitly capped the size of the workflow cache to 60 workflows. Eventually this number will need to be lowered as our codebase grows larger. I don't want to have to manually tune and tweak this value every few months. The workflow cache should be safely evicted from if it begins to get to the size where the process could crash as a result. I believe that when the cache grows too large, instead of crashing, old workflows should be evicted from the cache--particularly if the user has not changed the size of the cache themselves.

Minimal Reproduction

https://github.com/millerick/temporal-worker-crash-reproduction reproduces the issue

Follow the instructions in the README and a crash will happen in the worker

2026-07-21T18:26:14.282Z [ERROR] Worker failed {
  sdkComponent: 'worker',
  taskQueue: 'crash-repro',
  error: UnexpectedError: Workflow Worker Thread exited prematurely: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
      at Worker.<anonymous> (/System/Volumes/Data/src/temporal-crash-reproduction/node_modules/@temporalio/worker/lib/workflow/threaded-vm.js:68:30)
      at Worker.emit (node:events:518:28)
      at [kOnExit] (node:internal/worker:313:12)
      at Worker.<computed>.onexit (node:internal/worker:229:20)
      at Worker.callbackTrampoline (node:internal/async_hooks:130:17) {
    cause: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
        at [kOnExit] (node:internal/worker:313:26)
        at Worker.<computed>.onexit (node:internal/worker:229:20)
        at Worker.callbackTrampoline (node:internal/async_hooks:130:17) {
      code: 'ERR_WORKER_OUT_OF_MEMORY'
    }
  }
}
2026-07-21T18:26:14.282Z [INFO] Worker state changed { sdkComponent: 'worker', taskQueue: 'crash-repro', state: 'FAILED' }
[worker] FATAL — worker crashed: UnexpectedError: Workflow Worker Thread exited prematurely: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
    at Worker.<anonymous> (/System/Volumes/Data/src/temporal-crash-reproduction/node_modules/@temporalio/worker/lib/workflow/threaded-vm.js:68:30)
    at Worker.emit (node:events:518:28)
    at [kOnExit] (node:internal/worker:313:12)
    at Worker.<computed>.onexit (node:internal/worker:229:20)
    at Worker.callbackTrampoline (node:internal/async_hooks:130:17) {
  cause: Error [ERR_WORKER_OUT_OF_MEMORY]: Worker terminated due to reaching memory limit: JS heap out of memory
      at [kOnExit] (node:internal/worker:313:26)
      at Worker.<computed>.onexit (node:internal/worker:229:20)
      at Worker.callbackTrampoline (node:internal/async_hooks:130:17) {
    code: 'ERR_WORKER_OUT_OF_MEMORY'
  }
}
[worker][crash] rss=956MB (peak 2236MB, -1280MB) mainHeapUsed=209MB external=28MB

Environment/Versions

• OS and processor: observed on Mac and Linux
• Temporal Version: 1.17

Additional context

see above

[austin-tildei]

Ironically, we've been experiencing the same issue over the past week or two, after we fixed our issue related to #2166 and #2170. Our per-workflow-cache-entry size went up quite a bit after restoring proper isolation between them in the cache, resulting in OOMs as we fill up the cache, usually when bursts occur.

I've been playing around with estimating the cache entry size, and then calculating a reasonable max cache size at boot time, based on the available memory, but as you noted, this will need to adjust over time as the codebase grows over time, so it's good but not great.

I've also been thinking about extending the rust core package to expose an API to dynamically change the max cache size that we could invoke from the main TS context, in order to facilitate resizing at runtime based on current memory consumption, but I haven't gotten around to that just yet. Is that something the temporal team would be open to accepting as a PR into the upstream project?

For the sake of bookkeeping, I believe this is closely related to #1843.

[cconstable]

Thanks for writing this up and for the repro. Reproducing and investigating now.

[cconstable]

Had a chat with the team today about this and I've got discussion points and questions.

Where we are at

• To recap the issue: workflows caching can OOM workflow v8 isolates which the worker treats fatally. This causes the worker to shutdown and anything in-flight is redistributed potentially causing the issue to cascade.
• Workflow caching logic lives in both core (rust) and the typescript sdk. Core holds the runIds of all cached workflows (across all isolates) and the typescript side (the v8 isolates) hold the actual cached workflow.
• This is an issue we think should be addressed but prioritization is currently unclear. I've listed a few immediate remediations below.
• We would be open to proposls or PRs that address this and continuing the discussion here would be a great first start for that path.

Short-term fixes

capping maxCachedWorkflows

As you mentioned, this isn't ideal but is the easiest fix right now.

Reduce top-level state

Obligatory "reduce top-level state in workflow modules" e.g. can zod validation be shrunk down or live outside the workflow? I realize this is the least helpful "fix" but it's here for posterity.

preloadModules

Right now, some top-level modules can be shared using preloadModules. Give this a shot if applicable and see if it reduces memory footprint.

// via the Worker
Worker.create({
  workflowsPath: require.resolve('./workflows'),
  bundlerOptions: { preloadModules: [require.resolve('./shared-module')] },
});

// or if you pre-bundle
bundleWorkflowCode({
  workflowsPath: require.resolve('./workflows'),
  preloadModules: [require.resolve('./shared-module')],
});

Two caveats:

1. Preloaded modules are shared across all workflows, so they must hold no per-workflow mutable state
2. They are deep-frozen, so a module that lazily mutates its own internals at runtime can throw

Moving forward

Couple of options:

1. Allow the sdk to shrink the size of the cache. This could work but might not be that reliable depending on how it's implemented. The crux of the issue is the memory pressure of an individual isolate is what is causing the crashes, not the overall memory of the worker process and core doesn't know what cache entry is associated with what isolate
2. Add a new memory-aware cache + piggy back v8 isolate memory info back to core on existing calls (e.g. things that would be caching stuff)
3. Add a flag for reenabling the shared module behavior of [Bug] No isolation between worklflow modules with webpack >=5.108.0 #2166. This is unsafe but could be an escape hatch.

[cconstable]

Hopefully that is helpful context. Our primary typescript maintainer (@mjameswh) is out right now and I'm sure they will have some more opinionated thoughts on this when they return. If you'd like to take the path of submitting a patch for this we can continue chatting here too.

[millerick]

We've opted to cap the cache to a limited quantity, and recognize that if our code base we structured better there would be much less effect of top-level state and could potentially allow us to preload modules in a safe fashion.

We very much would like to see evictions happen rather than crashes--especially since Temporal's own estimates of how large the workflow cache should be are not accurate and allow the crashes to happen.

[cconstable]

We very much would like to see evictions happen rather than crashes

Very much agreed. Have this on the backlog. If anyone reading wants to get to it sooner feel free to assign yourself.

[danishaft]

@cconstable Could you assign me?

I can pick up the memory-aware cache side of this. I saw #2302, but that looks focused on limiting the workflow thread heap, so I don’t think we’d overlap. I’ll start with the repro here and look at passing the V8 thread’s memory usage back to Core so it can evict workflows before the thread OOMs.

[mjameswh]

Propagating operational metrics out of the workflow worker thread would definitely be helpful from an observability perspective. As it is now, there's no good way for operators to figure out the average memory footprint of their workflows.

However, I don't think that's the right avenue for the OOM issue. The problem is that if we depend on full round trip to Core for it to decide it’s time to evict some workflows, the worker thread will have OOM’ed long before a decision comes back.

The Worker Thread will have to make the decision by itself, and only notify Core that it dropped some workflows afterward. IIRC, lang-side can technically do that already, but I'm quite that code path isn't well exercised today, so there might be some rough edges to doing that, like possible race between lang-side eviction and an inbound WFT for a same run, I don’t know. Shouldn’t be prohibitively complex to resolve, though.

Another thing we should also consider is to automatically recreate the workflow worker thread if it dies on OOM. That’s basically the same as having lang side evict all cached workflows associated with that thread.

We're open to external contributions for some of these, but we'd need to narrow this down a bit.