Repository navigation
fix(pty): stop the non-blocking write losing input silently (#17381) - #17392
Conversation
`master_fd` is non-blocking (`os.set_blocking(..., False)` in `_spawn`), and the
write loop called `os.write(self.master_fd, data)` once, discarding the return
value, under a broad `except Exception: log; break`. That loses user input two
ways, both silent:
* `os.write` returns how many bytes it TOOK, and on a non-blocking descriptor
that can be fewer than given. The remainder was dropped, so a paste larger
than the PTY buffer arrived truncated -- at a byte boundary that can split a
multi-byte UTF-8 character.
* A full buffer raises `BlockingIOError`, which means "not now", not "failed".
The broad handler caught it, logged, and `break` -- ending the write thread
for the life of the session. The read loop keeps streaming output, so the
terminal looks healthy while accepting no input at all.
`_write_all` loops on the remainder and treats EAGAIN as backpressure.
THE BOUND IS TIME SINCE LAST PROGRESS, NOT TOTAL ELAPSED, and the difference is
a bug I shipped into the first version of this fix. A total budget looks
equivalent, passes every small-input test, and truncates exactly the large write
the method exists to deliver: a 1 MiB paste needs many buffer-fills and blows a
5s cap. My own test caught it -- it is the one test whose payload is large
enough to span multiple fills, which is why the payload size is now pinned in
its docstring. The clock resets on every byte accepted, so the budget detects a
child that has stopped reading rather than capping how long a legitimate write
may take.
On exhaustion the BlockingIOError is re-raised deliberately: a child that has
not accepted a byte in that long is wedged, and failing loudly beats retrying
forever.
Two smaller defects in the same file, from the same review:
* cleanup() closed the descriptor and set `master_fd = None` afterwards,
leaving a window where the write loop could use a closed fd -- and, once the
number was reused, one belonging to something else. It now publishes None
first and closes the fd it took.
* `TimingConstants.MICRO_DELAY` served both the write loop's idle bound and
the post-SIGTERM grace period. Unrelated timings sharing one constant means
tuning either silently moves the other, so the grace period is now
`_PROC_TERM_GRACE_S`. Same SSOT value today, separate names.
Tests run against a REAL non-blocking pipe, not a patched `os.write`: a fake
reproduces the semantics someone believed, while a pipe whose buffer is actually
full produces the real short write and the real EAGAIN and cannot drift from the
kernel. It also avoids patching a global pytest itself writes through.
Mutation checked: replacing `_write_all`'s body with a single `os.write` loses
the tail in the short-write test and raises in the backpressure test.
Closes #17381
|
Warning Review limit reachedNext included review available in 34 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ SSOT Configuration Compliance: Passing🎉 No new hardcoded values of either class — Known backlog in |
Thinking Path
master_fdis non-blocking —os.set_blocking(self.master_fd, False)in_spawn. The write loop then calledos.write(self.master_fd, data)once,discarded the return value, under a broad
except Exception: log; break.A non-blocking write expresses two outcomes that a single discarded call cannot:
os.writedoesBlockingIOErrormeaning buffer full, not nowbreak— ending the write thread for the session's lifeBoth lose user input silently. The second is the more visible failure and
the first is the worse one: the read loop keeps streaming output either way, so
the terminal looks healthy — but a truncated write corrupts quietly, at a byte
boundary that can split a multi-byte UTF-8 character, while a dead write thread
at least stops responding.
What Changed
autobot-backend/services/simple_pty.py_write_all();cleanup()fd-close ordering;_PROC_TERM_GRACE_Ssplit outautobot-backend/services/simple_pty_write_all_17381_test.py_write_allloops on the remainder and treatsEAGAINas backpressure.The bound is time since last PROGRESS, and that distinction was a bug I shipped
The first version of this fix bounded retrying by total elapsed time. That
looks equivalent to a stall bound, passes every small-input test, and truncates
exactly the large write the method exists to deliver — a 1 MiB paste needs many
buffer-fills and blows a 5s cap, so
_write_allre-raised and the data waslost. I would have reintroduced this very bug, for large inputs only.
It surfaced because one test uses a payload large enough to span multiple
buffer-fills; a 4 KB payload passes against both designs. That requirement is
now pinned in the test's docstring, because the test is worthless if someone
later shrinks the payload for speed.
_WRITE_STALL_BUDGET_Snow resets on every byte accepted, so it detects a childthat has stopped reading rather than capping how long a legitimate write may
take. On exhaustion it re-raises deliberately: a child that has not accepted
a byte in that long is wedged, and failing loudly beats retrying forever.
Two smaller defects in the same file, from the same review that found the first:
cleanup()— it closed the descriptor and setmaster_fd = Noneafterwards, leaving a window where the write loop could use a closed fd and,
once the number was reused, one belonging to something else. It now publishes
Nonefirst and closes the fd it took.MICRO_DELAYserved two unrelated timings — the write loop's idle boundand the post-
SIGTERMgrace period. One constant for two purposes meanstuning either silently moves the other. The grace period is now
_PROC_TERM_GRACE_S; same SSOT value today, separate names.Verification
Tests run against a real non-blocking pipe, not a patched
os.write. A fakecan only reproduce the semantics someone believed; a pipe whose buffer is
genuinely full produces the real short write and the real
EAGAINand cannotdrift from what the kernel does. It also avoids patching a global that pytest
itself writes through.
Mutation checked — replace
_write_all's body with a singleos.write(fd, data):BlockingIOErrorPre-push hook output is the evidence for the suite passing; a local run is not
CI's run, and CI is the authority.
Acceptance criteria
BlockingIOErrordoes not terminate the loop; data is retried and deliveredtest_a_full_buffer_is_backpressure_not_failuretest_a_multibyte_character_survives_a_split_writeexceptno longer catches retryable I/O; a fatal error still ends the loopThe caveat, stated rather than ticked silently. A
BlockingIOErrorthatsurvives a full
_WRITE_STALL_BUDGET_Swith no byte accepted is re-raised anddoes reach the broad handler, which still
breaks. That is deliberate: aftera full budget during which the child accepted nothing at all, the condition is
no longer retryable in any useful sense, and the alternative is a writer thread
that never returns. So the criterion is
met in the sense that matters — transient backpressure no longer kills the
session — but it is not literally true that the handler can never see an
EAGAIN. Reading it the literal way would require an unbounded retry, which isa worse bug than the one being fixed.
Closes #17381
Model Used
Claude Opus 5 (1M context).