You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
a self-update finishing at a logrotate boundary is reported FAILED after the full timeout #14804
A self-update finishing at a logrotate boundary is reported FAILED after an hour
autobot-slm-backend/api/code_sync.py:5573-5578
When _completion_is_from_a_rotated_log() is true, the watch declines to treat that
poll's completion as its own and keeps waiting. That is the right call in general —
the live log's mtime dates the truncation rather than a play, so the reading cannot
be trusted.
The gap is that the condition can become permanent for the remainder of the
watch. logrotate uses copytruncate and the running ansible process holds the
log fd open, so if the play's recap write lands right at the rotation boundary, the
recap ends up only in the rotated .1 file and the live log stays truncated — the
process has already exited, so nothing will ever write to it again for that run.
Every subsequent poll then sees the same rotated-source signal and keeps waiting. A genuinely successful self-update resolves only when _SELF_UPDATE_WATCH_TIMEOUT_SECONDS (3600s default) expires, and is then reported FAILED — a false negative on a deployment that actually worked.
Severity
Low. It is bounded — the deadline check runs unconditionally every iteration, so
there is no infinite hang — and the window is narrow, since rotation is daily
against a watch of at most an hour. But the outcome is wrong rather than merely
slow: an operator sees a failed self-update that succeeded, and the fleet stage does
not run.
Suggested fix
When the rotated source is detected, use the rotated file's own mtime as a secondary
freshness signal rather than deferring unconditionally: .1's mtime dates the actual
recap capture, so comparing it against since distinguishes "our recap, in the
rotated file" from "a stale reading we must not trust".
Acceptance criteria
A play whose recap lands only in the rotated file still resolves the stage
when that file's mtime is newer than stage.started_at
A genuinely stale rotated reading is still refused
A test covers both, and the refusing case fails if the guard is removed
The timeout still bounds the wait in every case
Found reviewing #14685, which is otherwise approved. Not introduced there — the
rotated-log guard is what makes the false negative reachable, and the guard itself
is correct and worth keeping.
CI on the merged head: 52 success, 7 skipped, zero failures; all 10 required checks green.
Each guard was mutation-checked with the target asserted present first, so a missing target fails loudly rather than reporting a false pass:
move delivery back after install -> CAUGHT
restore ignore_errors -> CAUGHT
drop admin-ui become -> CAUGHT
never return the rotated mtime -> CAUGHT
Still needed for #14809 to take effect on a live host
The code fix is landed but the deployed node still carries the stale requirements-ai.txt (81 lines, md5 f246c6af, spacy>=3.8.14 at line 32) against the repo's 98-line 07e3d1b3. The next update run is the one that delivers it, because the copy now happens before the install that used to abort first. Worth confirming on that run that stale_components finally drops autobot-ai-stack, rather than assuming the ordering fix was sufficient.
Activity
fix(slm): resolve the self-update stage when the play does not restar…
fix(update): deliver before installing, verify in a way that can fail (…
github-actions commentedon Aug 22, 2026
PR #14810 (merged to
Dev_new_gui) references this issue with a close keyword.If this issue is fully resolved, close it manually. If work remains, no action is needed.
mrveiss commentedon Aug 22, 2026
Delivered in PR #14810, squash-merged to
Dev_new_guias84c51d71a4a9ef2c720ca4afd7e1440e329ad780.Verified against the base tree, not the PR:
Deploy requirements-ai.txtnow precedes the filter and the install (task index asserted, not eyeballed), and the file is no longer copied a second time in the application-source loopignore_errorsgone) and retries rather than probing oncebecome: true_rotated_completion_time()present, the old boolean probe goneCI on the merged head: 52 success, 7 skipped, zero failures; all 10 required checks green.
Each guard was mutation-checked with the target asserted present first, so a missing target fails loudly rather than reporting a false pass:
Still needed for #14809 to take effect on a live host
The code fix is landed but the deployed node still carries the stale
requirements-ai.txt(81 lines,md5 f246c6af,spacy>=3.8.14at line 32) against the repo's 98-line07e3d1b3. The next update run is the one that delivers it, because the copy now happens before the install that used to abort first. Worth confirming on that run thatstale_componentsfinally dropsautobot-ai-stack, rather than assuming the ordering fix was sufficient.