Agentic Observatory post-rollout stabilization and audited-action readiness

[Svaag]

Context

The Agentic Observatory rollout is now live in production at https://observatory.servify.network/ in read-only mode.

Completed rollout milestones:

• Agent-Core Trace Pipeline rollout completed.
• Observatory North Star implementation completed across:
  • agent-core
  • noc-agent
  • engineering-loop
  • knowledge
  • agentic-observatory
  • network-operations
• Deployment PRs have been merged/applied:
  • feat: deploy Agentic Observatory #320 — initial Agentic Observatory deployment
  • chore: promote agent-core collector v0.5.0 #322 — Agent-Core collector v0.5 promotion for Observatory query APIs
  • chore: promote observatory HMAC fix #323 — Observatory HMAC canonicalization fix promotion
• Production verification completed:
  • DNS resolves for observatory.servify.network.
  • Caddy terminates TLS and proxies canonical host traffic.
  • agentic-observatory.service is active on loop.
  • /healthz returns 200 publicly and locally.
  • Authenticated read-only pages return 200:
    • /
    • /loops
    • /runs
    • /cases
    • /handoffs
    • /knowledge
    • /verification
    • /changes
    • /analysis
    • /api/loops/topology
  • Agent-Core collector query APIs return 200:
    • /v1/loops
    • /v1/runs
    • /v1/actions
    • /v1/topology
  • NOC Loop Console signed read APIs authenticate successfully.
  • State-changing Observatory actions are disabled and return 403.
• Current production mode:
  • OBSERVATORY_READ_ONLY=true
  • OBSERVATORY_ACTIONS_ENABLED=false

This issue tracks the next logical post-rollout work: close deployment drift, stabilize read-only operation, improve runbooks/monitoring, and prepare a safe audited-action canary without enabling actions prematurely.

Goals

1. Keep the production Observatory reliable and read-only while operators use it.
2. Eliminate rollout drift and rough edges found during deployment.
3. Document recovery/password/reset paths without exposing secrets.
4. Improve monitoring and alerting for Observatory and its dependencies.
5. Prepare—but do not prematurely enable—audited state-changing actions.

Non-goals

• Do not enable OBSERVATORY_ACTIONS_ENABLED=true as part of this issue unless a separate explicit approval/canary issue is completed.
• Do not bypass branch protection, CI, protected environments, or production deployment gates.
• Do not print or store Vault tokens, GitHub tokens, operator passwords, HMAC secrets, database passwords, or other secret material in logs/docs/issues.
• Do not expose the Observatory publicly beyond the existing operator-only authenticated console.

---

Phase 1 — Close deployment drift

1.1 Persist and apply trailing-dot Caddy canonicalization

A production hotfix was applied so copied FQDN URLs with the DNS root dot work correctly:

• https://observatory.servify.network./ → 301 https://observatory.servify.network/
• https://observatory.servify.network./healthz → 301 https://observatory.servify.network/healthz

Tracking PR:

• fix: redirect observatory trailing-dot host #326 — fix: redirect observatory trailing-dot host

Tasks:

• Review fix: redirect observatory trailing-dot host #326 normal PR comments.
• Review fix: redirect observatory trailing-dot host #326 inline review threads.
• Confirm fix: redirect observatory trailing-dot host #326 CI is green.
• Merge fix: redirect observatory trailing-dot host #326 through normal branch protection.
• Allow/apply the production deploy through the protected production environment.
• Verify Caddy on proxy after apply:
  • systemctl is-active caddy is active.
  • sudo caddy validate --config /etc/caddy/Caddyfile succeeds.
  • https://observatory.servify.network./ returns 301 to canonical no-dot host.
  • https://observatory.servify.network/healthz still returns 200 from the Observatory app.
• Confirm local Caddy IaC matches the live hotfix.

Acceptance criteria:

• fix: redirect observatory trailing-dot host #326 is merged and applied.
• No manual-only Caddy diff remains on proxy.
• Trailing-dot and canonical URLs both behave as expected.

---

Phase 2 — Stabilize read-only Observatory operation

2.1 Keep production read-only

Tasks:

• Confirm inventory/defaults continue to set:
  • agentic_observatory_read_only: true
  • agentic_observatory_actions_enabled: false
• Confirm rendered env on loop contains:
  • OBSERVATORY_READ_ONLY=true
  • OBSERVATORY_ACTIONS_ENABLED=false
• Confirm authenticated UI displays the read-only banner.
• Confirm action forms are not rendered where they should be hidden.
• Confirm a direct authenticated POST to an action endpoint still returns 403 Observatory actions are disabled.

Acceptance criteria:

• Operators can browse the Observatory.
• No state-changing action can be triggered from UI or direct POST while read-only mode is active.

2.2 Observe services for 24–72h

Watch for errors, restarts, memory growth, and upstream dependency failures.

Services/logs:

• agentic-observatory.service
• vault-agent-agentic-observatory.service
• agent-core-collector.service
• noc-agent.service Loop Console reads
• Caddy access/error logs for Observatory
• PostgreSQL on loop

Suggested checks:

ssh -i ~/.ssh/id_servify svag@2a0c:b641:b50:2::f0 \
  'systemctl is-active agentic-observatory.service vault-agent-agentic-observatory.service agent-core-collector.service postgresql'

curl -fsS https://observatory.servify.network/healthz

ssh -i ~/.ssh/id_servify svag@2a0c:b641:b50:2::f0 \
  'sudo journalctl -u agentic-observatory.service --since "24 hours ago" --no-pager | grep -E " 50[0-9] | 401 |Exception|ERROR" || true'

Acceptance criteria:

• No recurring 5xx pages.
• No unexpected Vault Agent auth/render failures.
• No sustained NOC Loop Console 401 errors for Observatory reads.
• No unexpected service restarts beyond planned deploys.

---

Phase 3 — Fix deployment/runbook rough edges

3.1 Document operator password reset flow

During rollout, password reset succeeded in Vault, but the rendered env initially did not update because the Observatory Vault Agent had consumed its wrapped SecretID and could not re-auth after restart:

error="no known secret ID"

Tasks:

• Add an Observatory operations runbook documenting password reset.
• Include the safe vault kv patch flow.
• Include a warning to use kv patch, not kv put, unless writing every existing key.
• Include a no-secret way to confirm rendered env state:

sudo awk -F= '/^OBSERVATORY_OPERATOR_USERNAME=/{print $0} /^OBSERVATORY_OPERATOR_PASSWORD_HASH=/{print "OBSERVATORY_OPERATOR_PASSWORD_HASH_present=yes length=" length($2)}' \
  /etc/agentic-observatory/observatory.env

• Document how to refresh the Observatory AppRole SecretID if Vault Agent reports no known secret ID.
• Document that secret values must not be pasted into issues, logs, shell history, PRs, or chat.
• Document service restart order:
  • Refresh Vault Agent AppRole credentials if needed.
  • Restart vault-agent-agentic-observatory.service.
  • Confirm env render mtime changed.
  • Restart or confirm agentic-observatory.service restarted.

Acceptance criteria:

• An operator can reset the Observatory password from the runbook without exposing secrets.
• The runbook covers Vault Agent re-auth failure recovery.

3.2 Fix systemd warning

Observed warning:

/etc/systemd/system/agentic-observatory.service:8: Unknown key 'ConditionPathIsReadable' in section [Unit], ignoring.

Tasks:

• Replace ConditionPathIsReadable with a supported systemd condition/assertion or remove it.
• Validate the rendered unit with systemd-analyze verify if possible.
• Apply and confirm warning no longer appears on daemon reload/restart.

Acceptance criteria:

• No ConditionPathIsReadable warning on daemon-reload or service restart.

3.3 Avoid app-promotion failure when Engineering Loop one-shot is active

A production apply initially failed because a one-shot hyrule-engineering-loop.service was running as user loop, causing Ansible user management to fail:

usermod: user loop is currently used by process ...

Tasks:

• Decide the intended behavior when the one-shot Engineering Loop is active during deploy:
  • pre-stop it,
  • skip user mutation when already correct,
  • retry after it exits,
  • or separate Observatory-only applies from full engineering-loop applies.
• Implement the chosen approach in Ansible/workflow.
• Add an IaC test if feasible.
• Verify rerun behavior when hyrule-engineering-loop.service is active.

Acceptance criteria:

• Observatory/app promotion deploys are not blocked by a currently running one-shot Engineering Loop process.

---

Phase 4 — Monitoring and alerting

4.1 Confirm blackbox/HTTP monitoring

Tasks:

• Confirm monitoring exists for https://observatory.servify.network/healthz.
• Confirm alerting route and severity are appropriate.
• Confirm Caddy proxy path is monitored, not only local app health.
• Confirm monitoring covers trailing-dot canonicalization if desired.

Acceptance criteria:

• A proxy/TLS/routing failure for the public Observatory URL pages operators.
• A local app-only failure is visible separately.

4.2 Add service-level checks

Tasks:

• Add/confirm Icinga service checks for:
  • agentic-observatory.service
  • vault-agent-agentic-observatory.service
  • local http://[loop]:8780/healthz
  • collector query API availability
• Add/confirm logs/metrics for Caddy 5xx on Observatory vhost.
• Add/confirm disk checks for Observatory state/log paths.

Acceptance criteria:

• Failures in the app, Vault Agent render, proxy, collector, or NOC dependency are observable.

---

Phase 5 — Read-only UX validation

Use the Observatory during real loop activity and collect UX/data issues before enabling actions.

Tasks:

• Validate /loops shows active Agentic Loops clearly.
• Validate /runs and run detail pages show action-by-action trace replay.
• Validate /cases and case detail pages show CaseService state clearly.
• Validate /handoffs shows LHP-v1 state clearly.
• Validate /knowledge shows artifacts and review state without treating operator text as model instructions.
• Validate /verification shows objectives/results clearly.
• Validate /changes and /analysis provide balanced better/worse impact assessment.
• Collect missing fields, confusing labels, slow pages, and broken links.
• Add small UI/data follow-up issues for each concrete gap.

Acceptance criteria:

• Operators can answer, from the Observatory alone:
  • What loops are active?
  • What action happened next in a run?
  • What case state exists in CaseService?
  • What handoffs exist and what state are they in?
  • What knowledge artifacts and verification objectives exist?
  • Did a change appear better, worse, or mixed?

---

Phase 6 — Prepare audited actions, but keep disabled

Actions should remain disabled until this preparation is complete and explicitly approved.

6.1 Define rollout order

Proposed order from lowest-risk to higher-risk:

1. Operator feedback/note
2. Case acknowledge
3. Case suppress with TTL
4. Verification result
5. Knowledge artifact review

Tasks:

• Confirm action rollout order.
• Define blast radius and rollback expectations for each action.
• Define which actions require extra confirmation.
• Define which actions are allowed only on test/canary cases first.

Acceptance criteria:

• Written action rollout plan exists before any production enablement.

6.2 Verify action safety contracts

For each action, verify:

• Authenticated server-side POST only.
• CSRF required.
• Authorization enforced.
• Idempotency key required/recorded.
• Target-scoped idempotency prevents cross-target reuse issues.
• CaseService/NOC validates current state.
• Audit/timeline entry recorded.
• Operator comments treated as untrusted text.
• No action accepts raw model-generated instruction as authority.
• Clear 403 behavior when read-only/actions-disabled.
• Tests cover success, duplicate idempotency, invalid CSRF, disabled action, and invalid target state.

Acceptance criteria:

• Every action has tests and a documented audit/timeline outcome.

---

Phase 7 — Canary audited actions separately

Create a separate issue/PR for actual action enablement after Phases 1–6.

Tasks for a future canary:

• Enable only the first low-risk action.
• Use a test/canary case or explicitly low-impact case.
• Keep all other actions disabled or hidden.
• Verify the NOC timeline/audit record.
• Verify idempotency prevents duplicate side effects.
• Verify rollback/expiry semantics if applicable.
• Observe for at least one full operator session before expanding.

Acceptance criteria for future canary:

• One action works safely with auditable, idempotent behavior.
• No duplicate or unexpected side effects occur.
• Operators agree the UI/action semantics are clear.

---

Suggested immediate next PRs

1. Merge/apply fix: redirect observatory trailing-dot host #326 — trailing-dot Caddy canonicalization.
2. Add docs/runbooks/agentic-observatory-operations.md covering:
   • login URL
   • read-only mode
   • password reset
   • Vault Agent AppRole recovery
   • health checks
   • common failures
3. Fix ConditionPathIsReadable warning in the systemd unit template.
4. Harden app-promotion deploy behavior around active hyrule-engineering-loop.service.
5. Add/confirm monitoring checks for public /healthz, local app health, and Vault Agent render freshness.

References

• Production URL: https://observatory.servify.network/
• Health URL: https://observatory.servify.network/healthz
• Initial deployment PR: feat: deploy Agentic Observatory #320
• Collector v0.5 promotion PR: chore: promote agent-core collector v0.5.0 #322
• Observatory HMAC promotion PR: chore: promote observatory HMAC fix #323
• Trailing-dot Caddy fix PR: fix: redirect observatory trailing-dot host #326
• Vault scope: kv/agentic-observatory
• AppRole: agentic-observatory
• Runtime host: loop
• Runtime port: 8780
• Initial production mode: read-only, actions disabled

[hyrule-engineering-loop]

Reliability Governor Decision

• role: staff_sre_autonomous_operations
• decision: knowledge_gap
• next_loop: knowledge
• handoff_contract: knowledge_context_pack
• source: knowledge
• intent: secret / tier 4
• capability: none
• knowledge: missing / unknown
• reasons: Knowledge policy result is deny; Knowledge context returned no included_refs

{
  "affected_assets": [
    "AS215932/network-operations"
  ],
  "affected_customers": [],
  "affected_services": [],
  "allowed_paths": [],
  "authority_text_hash": "a35c787d54643141ab4a37df617f6252447ab1867e9bc147c05f8afb92bea238",
  "blast_radius": "credential plane",
  "controlled_loops": [
    "engineering",
    "noc",
    "knowledge"
  ],
  "created_at": "2026-06-30T02:45:37.892656+00:00",
  "denial_reasons": [
    "Knowledge policy result is deny",
    "Knowledge context returned no included_refs"
  ],
  "expected_paths": [
    "secrets/"
  ],
  "forbidden_paths": [],
  "governor_name": "Reliability Governor",
  "governor_role": "staff_sre_autonomous_operations",
  "handoff_contract": "knowledge_context_pack",
  "intent_type": "secret",
  "issue_id": "AS215932/network-operations#327",
  "issue_number": 327,
  "issue_text_hash": "01c4084989ffe4370ed7883049b5f3a2d2fa1f71c89d95dfb40713a8957e65cc",
  "knowledge_authority_level": "unknown",
  "knowledge_context_pack_id": "ctx_5b659025e23ed8b8d86eee0ceda93b4e",
  "knowledge_export_version": "local-20260623120725",
  "knowledge_status": "missing",
  "labels_to_add": [
    "loop:knowledge-gap"
  ],
  "labels_to_remove": [
    "loop:intake",
    "loop:candidate",
    "loop:approved",
    "loop:needs-context",
    "loop:needs-human"
  ],
  "lhp": null,
  "matched_capability": null,
  "next_loop": "knowledge",
  "policy_rules": [
    "deny stale, contradictory, missing, or errored Knowledge context"
  ],
  "record_id": "60bf077c4ee0ad73cb07",
  "repo": "AS215932/network-operations",
  "required_checks": [],
  "risk_tier": 4,
  "rollback_plan": "Use the rollback/revert procedure specified in the request.",
  "routing_decision": "knowledge_gap",
  "schema_version": "reliability-governor.cdr.v1",
  "source": "knowledge",
  "storage_path": "/var/lib/engineering-loop/reliability-governor/as215932-network-operations-327-60bf077c4ee0ad73cb07.json",
  "verification_method": "Use the issue-specified verification: tests/checks/evidence named in the request."
}