bug(pty): SimplePTY._write_loop busy-spins on an empty queue; a leaked session degrades every later test in the shard #16828

Problem

SimplePTY._write_loop (autobot-backend/services/simple_pty.py:213-237) busy-spins with no backoff:

def _write_loop(self) -> None:
    while self.running and self.master_fd is not None:
        try:
            try:
                text = self.input_queue.get_nowait()
                ...
            except queue.Empty:
                continue

get_nowait() never blocks. On an empty queue this loop calls it again immediately, forever, at ~100% of one CPU core, for as long as self.running stays true and self.master_fd is set — i.e. for the whole life of any SimplePTY session whose write thread isn't explicitly stopped. (The sibling _read_loop does not have this problem — it polls with a 10ms timeout via poll_readable, per the #13219 fix.)

Evidence

python-suite shard 10/12 on PR #16789 (run 35128388669, job 104904838799) was cancelled at its 60-minute timeout-minutes ceiling after stalling at 98% test progress. Two faulthandler dumps (18:21:48Z and 18:51:30Z — 30 minutes apart) both show the main worker thread inside repo_tests/collected_test_model.py::test_functions_by_tree (via test_modules()'s repo-wide rglob), plus 6 live PTY threads — Thread-63/65/67 in _write_loop (simple_pty.py:220), Thread-62/64/66 in poll_readable (fd_poll.py:59) — present in both dumps.

Reproduced the suspect test in isolation locally (different Python/dep versions, so not full CI parity, but decisive for this question): pytest repo_tests/collected_test_model.py -k "test_functions_by_tree or test_modules" → 2 passed in 12.96s. This test is not intrinsically slow. Something in the full shard run left it starved.

Leading candidate for the leak: autobot-backend/api/simple_terminal_e2e_test.py::test_simple_terminal creates a real PTY-backed session via _create_session_via_rest() (line 75) with no visible .stop()/finally/fixture teardown in the file. By contrast, autobot-backend/tests/test_pty_read_loop_fd_setsize_13219.py's three tests all explicitly set session.running = False and reader.join(...) in a finally. I have not instrumented a run to confirm simple_terminal_e2e_test.py is the specific leak source — naming it as the strongest match to the observed thread signature, not a confirmed cause.

Why this matters beyond one cancelled run

ci.yml:131-136's own comment records "12m18s and 1h24m have both been recorded" for shard variance and sizes timeout-minutes: 60 against it as "generous headroom." If leaked spinning PTY threads are the actual mechanism, that variance is not random runner noise — it is however many CPU-bound leaked threads happen to accumulate before a slow/expensive later test (like the repo-wide rglob here) gets starved enough to matter. Raising the timeout or rebalancing the shard split would not fix this: a leak degrades every test that runs after it in the same worker process, regardless of which shard or how long the timeout is — it would just move where the wall gets hit, or hide it until a differently-unlucky shuffle.

Acceptance criteria

• SimplePTY._write_loop's queue.Empty branch backs off (short sleep, or block with a timeout like the read loop's poll_readable pattern) instead of spinning — defense in depth regardless of whether a caller ever leaks a session.
• simple_terminal_e2e_test.py's session-creating tests stop/tear down every SimplePTY/SimplePTYManager session they create, the same way test_pty_read_loop_fd_setsize_13219.py does.
• A guard (or at minimum a documented pattern) prevents a future PTY-session test from leaking its threads silently — nothing failed or errored here; the only symptom was unrelated tests slowing down later in the same worker.

Related

Found while root-causing #16789's shard 10/12 cancellation during today's fleet CI-hardening pass. ci.yml:131-136's variance comment may itself be describing this mechanism rather than genuine runner-speed noise — worth revisiting once this is fixed, to see if the "1h24m" tail disappears.