ThreadSettlementReactor spawns ~50 git/min even when auto-settle is disabled; vcs.refreshStatus sits at 10s p50

Summary

On a self-hosted server with many worktrees, the server spawns git processes
continuously for per-thread bookkeeping. The largest single contributor is
ThreadSettlementReactor.sweep, which performs its full pull-request lookup
before reading the settings that decide whether auto-settlement can happen at
all — so the work runs at full cost even with sidebarAutoSettleOnMerge: false and sidebarAutoSettleAfterDays: null.

The user-visible result is the "Some requests are slow — N requests waiting
longer than 15s" toast and repeated "… is reconnecting" banners, while the host
still shows idle CPU. Measured over 99.5 minutes of trace logs, the two RPCs
named in that toast have a median latency of ~10 seconds.

Environment

Note the settings row: automatic git fetch is off, and auto-settlement is
fully disabled. This is not the background fetch poller.

Measurements

Parsed from ~/.t3/userdata/logs/server.trace.ndjson*, attributing each
runGitCommand span to its root span by walking parentSpanId.

Window: 99.5 minutes, 11 rotated trace files.

runGitCommand spans: 4838  (49/min)

git spawns by root span:
  2286   ThreadSettlementReactor.sweep
  2103   http.server GET            (the /ws VCS status subscriptions)
   308   processRuntimeEvent
    79   (unknown)
    62   processDomainEvent

Completed-span latency (ms):

sql.execute is included to rule it out: the SQLite layer is not the
bottleneck. The sweep, at a 15.8 s median on a 60 s schedule, occupies
roughly a quarter of wall-clock time by itself.

Direct confirmation that settings do not gate the work. With
sidebarAutoSettleOnMerge: false and sidebarAutoSettleAfterDays: null, over a
180-second window:

git spawns since marker: 376
  190   ThreadSettlementReactor.sweep     <- 63/min, feature disabled
  186   (unknown)

Host-level effect — the box spends about twice as much CPU in the kernel as in
userspace, which is process creation rather than computation:

%Cpu:  19.5 us,  38.5 sy,  33.6 ni,  6.6 id
forks/sec: 19        context-switches/sec: 6868

Sampling the server's direct children for 5 seconds catches a continuous stream
of git processes and their zombies (47 [git] <defunct> in one sample).

V8 CPU profiles of the server's main thread (via --inspect,
Profiler.setSamplingInterval(1000)) during busy periods:

• 15–47% of samples in spawn
• ~18% in spanToTraceRecord / formatTraceExit / flushUnsafe — the
  server's own trace-record formatting
• ~5% in dedupeRemoteBranchesWithLocalMatches (2,465 remote refs)

Root cause

1. The sweep does its expensive work before checking whether it may act

From ThreadSettlementReactor (names as they appear in the published bundle):

const sweep = fn("ThreadSettlementReactor.sweep")(function* (mergedPullRequest) {
  const snapshot = yield* snapshots.getShellSnapshot();
  const candidates = snapshot.threads.filter((thread) =>
    isAutoSettlementCandidate(thread, now) && (...));
  const groups = Map.groupBy(candidates, lookupKey);
  // ...
  const pullRequest = yield* pullRequestFor(group[0]);   // <-- git + `gh pr list`
  yield* forEach(group, (thread) => gen(function* () {
    const settings = yield* settingsService.getSettings; // <-- settings read here
    const settledAt = resolveAutoSettlementAt({
      thread, pullRequest, now: decisionNow,
      autoSettleAfterDays: settings.sidebarAutoSettleAfterDays,
      autoSettleOnMerge: settings.sidebarAutoSettleOnMerge,
    });
    if (settledAt === null) return;                      // <-- always null when disabled

pullRequestFor → GitManager.branchPullRequest fans out into
GitVcsDriver.readConfigValue, resolveDefaultBranchName, remoteExists,
remoteBranchExists, isUnpublishedBranch, resolveRemoteRepositoryContext,
computeAheadCountAgainstBase, and finally gh pr list --head <branch> --state all. That is roughly 6–8 git spawns plus one gh invocation per candidate
group, every 60 seconds (ThreadSettlementReactor.start uses
repeat(spaced("1 minute"))).

When both settings are off, resolveAutoSettlementAt can only ever return
null, so 100% of that work is discarded.

2. Per-thread VCS status refresh scales with open threads

VCS_STATUS_REFRESH_INTERVAL = seconds(30) per subscribed thread, each
refresh running statusDetailsLocal, statusDetailsRemote,
resolveBranchHeadContext, computeAheadCountAgainstBase and friends. This is
the http.server GET bucket above — 2,103 git spawns, 43% of the total. It
correctly pauses when the client goes idle, but with a client open across many
threads it is the second-largest source.

3. Single-threaded amplification

All of this is orchestrated from one Node thread. Once spawn occupies a large
fraction of the event loop, interactive RPCs queue behind it — which is why the
symptom appears while the host still reports 60–70% idle CPU, and why reducing
process priority on other workloads does not resolve it.

What we ruled out

• SQLite — 77,301 sql.execute spans at p50 0 ms, p95 7 ms.
• Background git fetch — automaticGitFetchInterval: 0 throughout; setting
  it produced a large, separately measurable improvement and this is the
  remaining load on top of that.
• Host CPU saturation from other tenants — reproduced with the box at 60–70%
  idle and every competing workload at nice 19.
• GitHub rate limiting — 5000/5000 remaining; gh concurrency is already
  bounded at GITHUB_PROCESS_CONCURRENCY = 4.
• Stale remote refs — git remote prune --dry-run reports 0 prunable.

Suggested fixes

Roughly in order of value-to-effort:

1. Gate the sweep on its own settings. Return early when
   sidebarAutoSettleOnMerge === false && sidebarAutoSettleAfterDays === null,
   or hoist the settings read above pullRequestFor. This alone removes ~47% of
   git spawns for anyone who has auto-settle turned off.
2. Cache pull-request lookups per (repository, branch) with a short TTL.
   The sweep re-resolves identical branches every minute; PR state rarely
   changes that fast, and pullRequests.subscribeMerges already exists for the
   event-driven path.
3. Back off with scale. A fixed 60 s interval means cost grows linearly with
   thread count while the sweep's median duration is already 15.8 s. Consider
   scaling the interval by candidate count, or skipping when the shell snapshot
   is unchanged.
4. Batch the gh calls. One gh pr list --state all per repository, joined
   locally, instead of one invocation per branch.
5. Bound git spawning against the event loop. VCS_PROCESS_CONCURRENCY = 8
   bounds concurrency but not main-thread cost; moving spawn orchestration off
   the main thread would decouple interactive RPCs from bookkeeping.
6. Trace formatting cost. ~18% of main-thread samples in
   spanToTraceRecord/formatTraceExit seems high for always-on telemetry, and
   we could not find a setting or environment variable to reduce or disable it.
   Sampling, or moving formatting off the hot path, would help — as would simply
   documenting a way to turn it down.
7. Expose the interval. A threadSettlementSweepInterval alongside the
   existing backgroundActivity overrides would let large installations tune
   this without a patch.

Reproduction

1. Self-host the server against a repository with a large .git and many
   worktrees (ours: 26 worktrees, 1.6 GB .git, 2,465 remote refs).
2. Keep ~15 threads unsettled and unarchived.
3. Set sidebarAutoSettleOnMerge: false, sidebarAutoSettleAfterDays: null,
   and automaticGitFetchInterval: 0.
4. Watch ps for git processes parented to the server, or parse
   server.trace.ndjson* and attribute runGitCommand spans to their root
   span.

Expected: near-zero settlement-related git activity, since the feature is off.
Actual: ~63 git spawns per minute attributed to ThreadSettlementReactor.sweep.

Happy to supply raw trace files or .cpuprofile captures if useful.

Related issues

Searched before filing; this appears distinct from each of these:

• [Bug]: Background PR status polling via gh pr list drains GitHub GraphQL rate limit #3581 (closed) — Background PR status polling via gh pr list drains
  GitHub GraphQL rate limit. Same gh pr list fanout, but reported as a quota
  problem. This report adds the specific ordering defect (the settings that
  disable the feature are read after the lookup that costs the work) and
  measures the latency and CPU cost rather than quota use. On our install quota
  is not the constraint — 5000/5000 remaining while the symptom occurs.
• Overlapping VCS status refreshes multiply Git and network work #7076 (open) — Overlapping VCS status refreshes multiply Git and network
  work. Covers the explicit vcs.refreshStatus path and in-flight coalescing.
  That corresponds to the http.server GET bucket above (43% of git spawns);
  fixing it would not address the ThreadSettlementReactor.sweep bucket (47%),
  which runs on a timer with no client involved. The two are complementary.
• [Bug]: Local backend becomes CPU-bound and unresponsive during idle VCS/provider refreshes, causing reconnects #4773 (closed) — Local backend becomes CPU-bound and unresponsive during
  idle VCS/provider refreshes. Similar symptom class; this report isolates one
  specific remaining contributor with span-level attribution.