Prevent noc BGP router snapshots from filling root filesystem

[Svaag]

Incident context

The monitoring apply against noc failed in CI/CD at:

TASK [monitoring : Install node_exporter (Debian)]
Failed to update apt cache after 5 retries

Manual validation on noc showed the underlying apt error was disk exhaustion:

Error writing to file - write (28: No space left on device)
/dev/xvda1 20G 19G 0 100% /

The main growth source was unbounded router BGP snapshots:

/var/lib/hyrule-mcp/bgp-snapshots ~14G
bgp-router-snapshot.timer          hourly

The snapshot metadata includes expires_at at 7 days, but nothing on the host enforces retention. Once / filled, apt index refreshes failed and therefore unrelated Ansible applies failed.

Immediate mitigation already applied

On noc:

• Stopped and disabled bgp-router-snapshot.timer until retention is managed.
• Cleaned apt cache / journals.
• Verified apt-get update succeeds.
• Current state after mitigation:

/dev/xvda1 20G 7.4G 12G 40% /
/var/lib/hyrule-mcp/bgp-snapshots 3.7G
bgp-router-snapshot.timer disabled/inactive
apt-get update OK

Engineering Loop task

Implement a permanent, reviewed fix so BGP snapshots cannot fill noc again, then safely re-enable collection.

Preferred Network-Operations-side approach:

1. Manage bgp-router-snapshot.service and bgp-router-snapshot.timer in Ansible, probably in ansible/roles/hyrule_mcp or the noc play path, instead of leaving them as manual drift.
2. Add enforced local retention for /var/lib/hyrule-mcp/bgp-snapshots before re-enabling the timer. Acceptable implementations:
   • systemd-tmpfiles policy for age-based deletion; or
   • a small managed cleanup service/timer; or
   • call a hyrule-mcp retention flag if that exists/gets added.
3. Keep the retention window aligned with snapshot metadata (expires_at currently 7 days), or make it an Ansible default such as hyrule_mcp_bgp_snapshot_retention_days: 7.
4. Add monitoring/guardrail if not already sufficient: root filesystem disk checks should alert before apt/apply breaks.
5. Re-enable bgp-router-snapshot.timer only after retention is active.

If the cleanest fix requires code changes in AS215932/hyrule-mcp, file/link a child issue or PR there, but this Network-Operations issue is the production rollout tracker.

Acceptance criteria

• Ansible owns the BGP snapshot timer/service or explicitly removes/disables unmanaged copies.
• Retention is enforced automatically on noc for /var/lib/hyrule-mcp/bgp-snapshots.
• The retention horizon is documented/configurable and defaults to 7 days.
• Applying playbooks/noc.yml (or the chosen owning playbook) is idempotent.
• After apply on noc:
  • systemctl is-active bgp-router-snapshot.timer is active.
  • snapshot cleanup policy/timer exists and is active, or tmpfiles policy is installed and systemd-tmpfiles --clean succeeds.
  • df -h / has healthy free space.
  • apt-get update succeeds.
• The PR summary includes the incident root cause and the manual mitigation that disabled the timer.

Suggested validation commands

ansible-playbook ansible/playbooks/noc.yml --tags apply -e '{"noc_apply":true}' --limit noc
ssh noc 'systemctl status bgp-router-snapshot.timer --no-pager'
ssh noc 'systemd-tmpfiles --clean || true'
ssh noc 'du -sh /var/lib/hyrule-mcp/bgp-snapshots; df -h /; apt-get update'

[hyrule-engineering-loop]

Reliability Governor Decision

• role: staff_sre_autonomous_operations
• decision: knowledge_gap
• next_loop: knowledge
• handoff_contract: knowledge_context_pack
• source: human
• intent: monitoring / tier 1
• capability: none
• knowledge: missing / unknown
• reasons: Knowledge policy result is deny; Knowledge context returned no included_refs

{
  "affected_assets": [
    "AS215932/network-operations"
  ],
  "affected_customers": [],
  "affected_services": [
    "monitoring"
  ],
  "allowed_paths": [],
  "authority_text_hash": "098acbdba696d62c6ce2d015428bfd6cdea68fe3be9f1bb2b1bd0d01cc6267d0",
  "blast_radius": "operator monitoring",
  "controlled_loops": [
    "engineering",
    "noc",
    "knowledge"
  ],
  "created_at": "2026-06-30T01:54:52.386705+00:00",
  "denial_reasons": [
    "Knowledge policy result is deny",
    "Knowledge context returned no included_refs"
  ],
  "expected_paths": [
    "docs/",
    "monitoring/",
    "alerts/",
    "tests/"
  ],
  "forbidden_paths": [],
  "governor_name": "Reliability Governor",
  "governor_role": "staff_sre_autonomous_operations",
  "handoff_contract": "knowledge_context_pack",
  "intent_type": "monitoring",
  "issue_id": "AS215932/network-operations#321",
  "issue_number": 321,
  "issue_text_hash": "ab2171204b546690fa554fb460f155e7d522f66aa06ca42ac1200550f5250efb",
  "knowledge_authority_level": "unknown",
  "knowledge_context_pack_id": "ctx_37b919c272e6e9dea86061669c22a5eb",
  "knowledge_export_version": "local-20260623120725",
  "knowledge_status": "missing",
  "labels_to_add": [
    "loop:knowledge-gap"
  ],
  "labels_to_remove": [
    "loop:intake",
    "loop:candidate",
    "loop:approved",
    "loop:needs-context",
    "loop:needs-human"
  ],
  "lhp": null,
  "matched_capability": null,
  "next_loop": "knowledge",
  "policy_rules": [
    "deny stale, contradictory, missing, or errored Knowledge context"
  ],
  "record_id": "222c6fd31f26ed8f7349",
  "repo": "AS215932/network-operations",
  "required_checks": [],
  "risk_tier": 1,
  "rollback_plan": "Revert the draft PR; for deployed alert tuning, restore the previous rule/config version.",
  "routing_decision": "knowledge_gap",
  "schema_version": "reliability-governor.cdr.v1",
  "source": "human",
  "storage_path": "/var/lib/engineering-loop/reliability-governor/as215932-network-operations-321-222c6fd31f26ed8f7349.json",
  "verification_method": "Use the issue-specified verification: tests/checks/evidence named in the request."
}