Skip to content

a self-update finishing at a logrotate boundary is reported FAILED after the full timeout #14804

Closed

Description

@mrveiss

A self-update finishing at a logrotate boundary is reported FAILED after an hour

autobot-slm-backend/api/code_sync.py:5573-5578

When _completion_is_from_a_rotated_log() is true, the watch declines to treat that
poll's completion as its own and keeps waiting. That is the right call in general —
the live log's mtime dates the truncation rather than a play, so the reading cannot
be trusted.

The gap is that the condition can become permanent for the remainder of the
watch
. logrotate uses copytruncate and the running ansible process holds the
log fd open, so if the play's recap write lands right at the rotation boundary, the
recap ends up only in the rotated .1 file and the live log stays truncated — the
process has already exited, so nothing will ever write to it again for that run.

Every subsequent poll then sees the same rotated-source signal and keeps waiting. A
genuinely successful self-update resolves only when
_SELF_UPDATE_WATCH_TIMEOUT_SECONDS (3600s default) expires, and is then reported
FAILED — a false negative on a deployment that actually worked.

Severity

Low. It is bounded — the deadline check runs unconditionally every iteration, so
there is no infinite hang — and the window is narrow, since rotation is daily
against a watch of at most an hour. But the outcome is wrong rather than merely
slow: an operator sees a failed self-update that succeeded, and the fleet stage does
not run.

Suggested fix

When the rotated source is detected, use the rotated file's own mtime as a secondary
freshness signal rather than deferring unconditionally: .1's mtime dates the actual
recap capture, so comparing it against since distinguishes "our recap, in the
rotated file" from "a stale reading we must not trust".

Acceptance criteria

  • A play whose recap lands only in the rotated file still resolves the stage
    when that file's mtime is newer than stage.started_at
  • A genuinely stale rotated reading is still refused
  • A test covers both, and the refusing case fails if the guard is removed
  • The timeout still bounds the wait in every case

Found reviewing #14685, which is otherwise approved. Not introduced there — the
rotated-log guard is what makes the false negative reachable, and the guard itself
is correct and worth keeping.

Activity

  1. github-actions commented on Aug 22, 2026

    @github-actions
    Contributor

    PR #14810 (merged to Dev_new_gui) references this issue with a close keyword.

    fix(update): deliver before installing, verify in a way that can fail (#14809, #14701, #14700, #14804)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  2. mrveiss commented on Aug 22, 2026

    @mrveiss
    OwnerAuthor

    Delivered in PR #14810, squash-merged to Dev_new_gui as 84c51d71a4a9ef2c720ca4afd7e1440e329ad780.

    Verified against the base tree, not the PR:

    CI on the merged head: 52 success, 7 skipped, zero failures; all 10 required checks green.

    Each guard was mutation-checked with the target asserted present first, so a missing target fails loudly rather than reporting a false pass:

    move delivery back after install -> CAUGHT
    restore ignore_errors            -> CAUGHT
    drop admin-ui become             -> CAUGHT
    never return the rotated mtime   -> CAUGHT
    

    Still needed for #14809 to take effect on a live host

    The code fix is landed but the deployed node still carries the stale requirements-ai.txt (81 lines, md5 f246c6af, spacy>=3.8.14 at line 32) against the repo's 98-line 07e3d1b3. The next update run is the one that delivers it, because the copy now happens before the install that used to abort first. Worth confirming on that run that stale_components finally drops autobot-ai-stack, rather than assuming the ordering fix was sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions