fix(rollback): preserve the executing reconcile unit and its systemd timeout budget #93

Problem

Rollback removes managed systemd unit files, including the reconcile service that may currently be executing, and calls systemctl daemon-reload before restoring checkpointed units. This exposes the running reconciliation to systemd defaults instead of its configured timeout budget and can interrupt recovery.

The September 8 operator diagnostic attributes an earlier 90-second timeout to this sequence. Installed reconcile settings were TimeoutStartSec=2h and TimeoutStopSec=15min; the systemd manager default was 90 seconds. The latest two approximately 128-second reconciliation attempts completed checkpoint restoration before the reported race fired. Their immediate failure was the separate Docker network health defect tracked in #92, not the two-hour service timeout. Runtime observations are operator-supplied; the remove/reload/restore sequence was independently verified in source.

Verified source

scripts/install-worker-controller.sh at PR #91 head includes ci-fleet-reconcile.service in optional_unit_names. remove_systemd_units() deletes that array's unit files and immediately reloads systemd; checkpoint systemd restoration uses that removal path before restoring the saved units. The same unsafe sequence exists on main.

Scope

Make checkpoint unit restoration safe while running inside the reconcile service. Keep the executing service definition available across restoration and ensure any daemon reload observes a complete valid unit set with the intended timeout settings. Separate rollback restoration from uninstall deletion as necessary. Preserve checkpoint recovery, timer state restoration, and the existing transaction-result contract.

Acceptance criteria

• Rollback never presents systemd with the currently executing reconcile unit removed during a daemon reload, and never stops/restarts its own parent service to restore the checkpoint.
• Effective start/stop timeout settings remain valid throughout rollback; they do not transiently revert to manager defaults.
• Deterministic regression coverage checks unit presence and contents at each daemon reload, restored timer enabled/active states, and both successful and failed checkpoint restoration.
• A timing regression models or exercises recovery exceeding the manager default timeout but within the configured service budget. Record whether evidence comes from a model or a real disposable systemd environment; mocks alone must not be described as proof of real systemd timing behavior.
• Explicit uninstall retains its intended cleanup semantics; normal install/upgrade and zero-runner drain behavior remain covered.
• Recovery failures remain explicit and preserve recovery evidence; no false success is emitted.
• Exact implementation head passes Validate ci-fleet prototype / Build without registering a runner, including scripts/validate.sh, shell checks, and rollback/reconciliation regressions.

No arbitrary timeout increase, permanent host override, guaranteed recovery after process death/reboot, live fault injection, or production deployment is included in this code fix.

Coordination and rollout

Coordinate with #91, which already modifies the reconciliation/installer transaction. Reviewed PR head when filing: 779e69f9aaecbcd43e465629264ca181cfa7b3af; main/base: 003de44f7e27d7bfe5bb753098efb6eb5f9712f1. Avoid overlapping installer writers; refresh the PR head before implementation.

After the reviewed fixes merge, operational rollout needs a new private configuration commit selecting the fixed engine. Reapplying an existing configuration pinned to the old engine will not deploy the fix. Keep private configuration and infrastructure records outside this public issue. Repository completion and subsequent operational verification should be reported separately.

Operational follow-up should verify successful scheduled reconciliation after rollout. Any deliberate rollback exercise belongs in an isolated disposable environment; do not induce a production failure to close this repository issue.