Skip to content

[DESIGN] The store keeps no tombstone for a removed server, so file/store reconciliation cannot tell 'never added' from 'deliberately removed' #2258

Closed

Description

@erikdarlingdata

Split out of #2254's review, which caught the ambiguity while the diagnostic was being written.

The ambiguity

The Viewer's Remove action hard-deletes the registry row:

DELETE FROM config_monitored_servers WHERE server_id = $1

and deliberately never edits darling.json — the file is a one-time bootstrap, and SeedMonitoredServersAsync's own comment records the intent ("a later Viewer deletion (Stage 3) is never resurrected by a re-seed").

So a server named in the file but absent from the store has two indistinguishable causes:

  1. it was added to the file after the first seed and was never registered — the operator expects it to be monitored and it is not ([QUESTION] Add new server to monitoring #2252, the field report)
  2. it was registered once and then deliberately removed via the Viewer, and nobody bothered to edit the file — a correct state

The store retains nothing that separates them. #2254's reconciliation line therefore reports at Information and words itself for both, which is honest but weaker than it could be: cause 1 deserves a warning, and cause 2 deserves silence.

Fix shape

A tombstone — the minimum being the server_id of anything removed, written by the same code path that deletes the row. Then reconciliation becomes exact: in the file, absent from the store, and not tombstoned means cause 1 and can warn; tombstoned means cause 2 and stays quiet.

Design questions worth settling before building it, because they decide whether the tombstone helps or becomes its own trap:

  • Where. A config_removed_servers table, or an is_removed flag on a soft-deleted row. A flag keeps the identity and its history in one place; a separate table keeps the registry clean and cannot be confused with is_enabled = FALSE (which today means "registered but paused", a different state that must keep collecting nothing while remaining monitored-in-principle).
  • Re-adding. Adding a tombstoned server back must clear the tombstone, or the reconciliation goes quiet about a server that genuinely is not registered the next time it is dropped from the store by other means.
  • Retention. A tombstone is tiny and permanent by nature. Worth deciding whether it survives a store rebuild, since a rebuilt store re-seeds from darling.json and would resurrect exactly the servers the tombstones exist to remember.
  • Whether it should exist at all. The alternative is to accept the ambiguity permanently and keep the Information line. Given the accumulation is one row per server ever removed, the cost is negligible; the question is whether the warning/silence distinction is worth a schema rung.

Severity

Cosmetic. The reconciliation line is already correct for both cases — this only lets it be sharper. Not worth a rung on its own; worth folding into the next migration that touches config_monitored_servers, e.g. whatever #2218 (identity carries neither engine nor port) ends up doing to that table.

Activity

  1. erikdarlingdata commented on Aug 15, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed in #2285, merged to dev — without the tombstone table or the rung this issue proposed, because the fact a tombstone would record already exists.

    collect.servers is the OBSERVED registry: it gets a row on every successful connect, the Viewer's Remove deletes only from config_monitored_servers (the DESIRED config), and nothing purges it — it is a registry rather than a time series, which I checked against the retention purge list rather than assumed. So "was this server ever really monitored" is answerable today, with no schema change and no second piece of state that could drift from the first.

    The reconciliation is now two lines: a warning for servers never monitored (the #2252 case, where something the operator wants is not happening) and an Information line for deliberately-removed ones that advises nothing and notes their collected history is kept.

    Answering the design questions you raised:

    • Where — neither. The is_removed flag was worth rejecting explicitly: is_enabled = FALSE already means "registered but paused", so a second flag makes (is_enabled, is_removed) a four-state space with two meaningless combinations, and every existing reader of that table would have to learn the new flag or silently start including removed servers. That is the same seam failure [BUG] Registration-time collision check: refuse a new registration whose host + actual database matches an existing one #2280 had to fix in the dedupe gate.
    • Re-adding — no tombstone to clear, so the concern dissolves.
    • Retention / store rebuild — a rebuilt store has no observed rows, so everything reads as never-monitored until it connects once. That degrades to a warning rather than to silence, which is the safe direction where the file genuinely is the declared intent. Documented on the method.
    • Whether it should exist at all — the distinction was worth having; a rung for it was not.

    One limit stated on the method rather than left to be discovered: a server registered but never successfully connected to reads as never-monitored. That is the right answer to "is this being monitored?" even though it is the wrong answer to "was it ever registered?" — and the operator's question is the former.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions