Skip to content

[Bug]: thread.session.stop is a no-op once the thread is archived, so rapid stop+archive leaks provider processes #14203

Description

@josephv123

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Uses the same HTTP orchestration API the web client uses (POST /api/orchestration/dispatch), so this can be hit by any client that archives right after stopping (or tears down several threads at once).

Deterministic (archive before stop):

  1. Start a thread on any provider (seen with Claude Code and Codex) and let the first turn finish, so the session is ready and a claude / codex app-server process is alive.
  2. Dispatch {"type":"thread.archive","threadId":…}, then immediately {"type":"thread.session.stop","threadId":…}.
  3. Wait 30s and check the provider process.

Realistic (bulk teardown):

  1. Start 4 threads (any providers) and let them reach ready.
  2. For each thread in turn, without waiting in between, dispatch thread.turn.interrupt, then thread.session.stop, then thread.archive.
  3. Wait 30s and check the provider processes.

Expected behavior

thread.session.stop stops the provider session and its process whether or not the thread is archived, or the archive stops the session itself. An archived thread should never keep a live provider process.

Actual behavior

  • Deterministic case: the provider process keeps running indefinitely (1/1).
  • Bulk case: only the first thread's session is stopped. The other 3 of 4 provider processes keep running with no visible thread in the UI.

In load testing, a 10-thread teardown leaked 9 of 10 provider processes (Claude Code + Codex, about 2.4 GB RSS on a 6.6 GB machine), which stayed until I killed them by hand. In a 24-thread teardown, only 1 of 24 stop commands actually stopped a session.

The server trace shows the stop commands are received but not acted on. The first stopSessionInternal takes ~494 ms; the remaining 23 processSessionStopRequested spans are then drained in the same millisecond (~0.2 ms each) with no corresponding stopSessionInternal. By then those threads' thread.archive commands had already been dispatched. It looks like the stop reactor skips (or can't resolve) the session once the thread is archived, so any stop that is still queued behind a slow stop is silently dropped.

Control: stop, wait ~3s, then archive → process exits every time.

Impact

Major degradation or frequent failure

Version or commit

t3 v0.0.42

Environment

Ubuntu, Linux 7.0.0, headless t3 serve (background service). Providers: Claude Code 2.1.283 (claudeAgent), codex-cli 0.157.1 (codex app-server).

Logs or stack traces

# server.trace.ndjson, bulk teardown of 24 threads (interrupt → stop → archive per thread)
1790647304.85 stopSessionInternal            493.6ms   # thread 1: actually stopped
1790647305.34 processSessionStopRequested      0.5ms   # threads 2..24: all drained here,
1790647305.34 processSessionStopRequested      0.2ms   #   no stopSessionInternal follows
...           (22 more, all ~0.2ms)
1790647310.71 stopSessionInternal              0.1ms   # these only appear after I SIGTERM'd
...           (10 more)                                #   the leaked processes by hand
# counts in the window: 24 thread.session.stop commands, 24 processSessionStopRequested, 1 real stopSessionInternal

Workaround

Wait for the session to reach stopped before dispatching thread.archive, and tear threads down one at a time. Leaked processes have to be killed manually (they're children of the t3 serve process, with cwd = the thread's worktree/project).

Activity

  1. t3dotgg commented on Oct 2, 2026

    @t3dotgg
    Member

    Note

    🤖 GPT-6.1-Sol responding on behalf of Theo

    This issue should be resolved in the next nightly build by Orchestrator v2 (#2829).

    Please try that nightly. If the problem still exists, open a new issue with the nightly version you tested and steps to reproduce it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions