Skip to content

bug(deploy): the SLM frontend promotion is two renames, so the served directory briefly does not exist #15610

Description

@mrveiss

Found reviewing PR #15602, which fixed #15557's real defect and carried this one forward.

Problem

The promotion step in roles/_shared/tasks/build_publish_slm_frontend.yml is two sequential renames:

mv dist          -> dist.previous
mv dist.staging  -> dist

Between them dist/ does not exist on disk. A request landing in that window gets a 404/403 — not the previous bundle and not the new one.

Scope and honesty about severity

This is not a regression introduced by #15602. It is the pre-existing code from update-all-nodes.yml, lifted verbatim into the shared file. #15602 is a behaviour-preserving extraction and is correct as such.

But it changes the exposure: the window used to exist at one entry point and now exists at four, because the extraction is what made all four consistent. Consistency was the goal, and it worked — this is the cost of having got it.

Nor is it the defect #15557 was about. That one was a failed build leaving a broken site serving indefinitely, which the staging directory and the non-empty index.html check now genuinely fix. This window is milliseconds on a successful publish. It is a smaller problem than the one that was solved, and it should be recorded rather than left implicit in a review comment.

Fix

A directory rename pair cannot be atomic. A symlink flip can:

dist-<build-id>/          # each build lands in its own directory
current -> dist-<id>      # ln -sfn, a single atomic syscall

rename(2) on the symlink itself is atomic, so no request ever observes an absent target. The served path becomes current/.

Acceptance criteria

  • The served path is switched by a single atomic operation; no window exists in which it resolves to nothing
  • Old build directories are retained for rollback and pruned on a bounded policy, so the disk does not grow without limit
  • The web server config points at the stable path, and the change lands with it rather than after it
  • All four entry points get it together — they share one task file now, so this is one edit rather than four, which is the property fix(ansible): stage the SLM frontend publish and repair three backend path defects (#15557, #15560) #15602 bought
  • A test proves the target is never absent: assert the promotion is a single step, not a sequence that passes through an unresolvable state

Activity

  1. github-actions commented on Sep 4, 2026

    @github-actions
    Contributor

    PR #15654 (merged to Dev_new_gui) references this issue with a close keyword.

    fix(deploy): publish the SLM frontend by flipping a symlink, not by two renames (#15610)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  2. mrveiss commented on Sep 4, 2026

    @mrveiss
    OwnerAuthor

    Closed on code evidence, verified against merged Dev_new_gui (PR #15654, 450ac0a3).

    AC Evidence from base
    The served path is switched by a single atomic operation current staged as .current.next then replaced with one mv -T / os.replace. No window in which the name resolves to nothing
    Old builds pruned on a bounded policy Newest 3 kept, bound in an inventory variable and an env-backed Python constant
    Rollback still works previous replaces dist.previous; rolling back is the same single syscall
    The web server points at the stable path Both nginx templates and the systemd probe name current
    All four entry points together All four now render the site config as well as publishing
    A test proves the target is never absent repo_tests/slm_frontend_atomic_publish_15610_test.py, asserting the behaviour by spying on every os.replace/unlink/rmtree

    Two things the work found that the issue did not describe

    The self-update path never rendered the site config. update-all-nodes.yml PLAY 1 deliberately does not run the slm_manager role, so it would have published to a path nginx never opened — a green deploy serving the old bundle.

    And so did update-node.yml, which review caught after the first fix. The shared task file's own header names four publishing entry points; the first pass wired two, a third self-updates, and this one was missed. Every already-provisioned node still had an nginx config naming dist/, so the next run against any of them would have published a fully-verified bundle to current while nginx served the stale one — silently, because every check inside the publish task file passes.

    That is the fixed in three places, left in one pattern, and it is why the closing evidence above counts all four rather than confirming the mechanism works.

    Two design details worth keeping

    Retention excludes the current/previous targets by name rather than trusting them to fall inside the newest-3 window — a rollback moves current backwards, and a window rule would eventually delete the thing you rolled back to. Pinned by a test that deliberately places a rolled-back current outside the window.

    The Python self-sync publisher was updated in the same change. Leaving it on dist would have had one publisher writing where nothing serves, and the node's serving path would depend on which publisher last ran.

    One health-probe fix

    A current that exists as a symlink but resolves to nothing failed is_dir() and landed in not_applicable — reported as "this node serves no UI" and dropped from health rollups, when the node was publishing and its target had gone. Now unhealthy, with a contrast pair so a genuinely backend-only node still reports not_applicable.

    Filed, not folded in

    #15648, #15649, #15650, #15653, #15659 — the legacy dist/ compatibility path, an entry point that publishes without rendering, a bootstrap script using the wrong npm target, mid-session client 404s on hashed assets, and a sync script publishing via an inventory path that does not exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions