feat: deploy shared LHP-v2 agentic coordination plane

[Svaag]

Outcome

Ships the dark-by-default production scaffold for the organization-wide LHP-v2 coordination plane and dedicated SOC Agent VM.

What changes

• deploys the central signed Agent Core coordinator on loop with Postgres, Vault, overlay firewalling, health monitoring, and exact-SHA apply gates;
• wires NOC, Engineering, Knowledge, SOC, and the Agentic Observatory to the same coordinator contract, with independent per-loop Vault keys;
• provisions the dedicated soc inventory/DNS/NoCloud/logging/monitoring/firewall surface;
• adds the six-rung SOC rollout ladder through exact-scope, individually senior-approved bounded RT-2 probes, without remediation credentials;
• stages Knowledge coordinator intake as reviewed A4 learning proposals feeding the existing Knowledge Loop PR path;
• extends Observatory GitHub OAuth/org/team/2FA configuration and coordinator views/actions, all separately gated;
• extends post-merge promotion dispatch and exact-SHA pinning to Agent Core, SOC, NOC, Engineering, Knowledge, and Observatory;
• applies destination firewalls before the first SOC promotion and preserves all new workers/services disabled until reviewed rollout stages.

Deployment order

1. Merge this infrastructure scaffold first; moving refs remain non-deployable and all new services are disabled.
2. Merge the six app PRs after green CI. Their request-promotion workflows coalesce exact SHAs into the normal promotion PR.
3. Bootstrap the scoped Vault entries and GitHub OAuth/policy identity.
4. Follow docs/runbooks/agentic-coordination-shadow-cutover.md one reviewed stage at a time.

No production apply or mode promotion is performed by this PR.

Validation

• scripts/ci/iac-static.sh — 109 tests, DNS parsers, generated-flow freshness, deploy preflight: pass
• affected Engineering, NOC, and SOC Ansible syntax checks: pass
• Engineering and SOC local inventory renders: pass
• Ruff check/format for promotion scripts and new tests: pass
• yamllint: pass (warnings only under repository policy)
• ansible-lint: pass at configured profile (existing warnings only)

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 2026c7c)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 5 🔵🔵🔵🔵🔵

🏅 Score: 85

🧪 PR contains tests

🔒 Security concerns Yes: The following security concerns were identified. 1) Vault AppRole credentials (role_id, wrapped_secret_id) are injected into the job environment via $GITHUB_ENV with ::add-mask::, which is non-retroactive. If any downstream step prints these variables (e.g., via set -x or an error traceback), they will appear unmasked in the workflow log. 2) git rev-parse origin/main is used to determine the network-operations pin in the SOC promotion path, which does not pin to a reviewed SHA and can result in deploying an unreviewed revision on a race condition. 3) In the Agent Core coordinator role, the Vault-rendered database password is written to a temporary SQL file on disk before being piped to psql. If the playbook is interrupted before cleanup runs, password material remains on disk. 4) The custom approach to piping Vault-rendered secrets into psql (via Python parsing a file) is fragile and may leak passwords in error messages. 5) Permission 0640 on Vault-rendered secret files (e.g., soc-agent.env.ctmpl.j2 rendered to soc_agent_secrets_env_file) is reasonable, but the SOC playbook reuses the same reload command for all timers (handoffs, probes, posture) which could cause unnecessary service restarts or masked failures if one timer is not yet installed. 6) The ansible.builtin.shell task that parses the secrets env file uses no_log: true correctly, but the inner Python snippet runs print(password) which could spill into ansible stdout if not suppressed.

⚡ Recommended focus areas for review Vault Secret in Job Output The 'Mint agent-core-coordinator Vault bootstrap' and 'Mint SOC Agent Vault bootstrap' steps write the role_id and wrapped_secret_id to $GITHUB_ENV via printf and mark them with ::add-mask::. However, ::add-mask:: is not retroactive—if any later step echoes these variables directly or indirectly (e.g., by printing the environment or passing them to a subsequent shell that has set -x), the secret may appear unmasked in runner logs. Additionally, the variables are exported broadly across subsequent job steps via $GITHUB_ENV where they could be accidentally dumped in error messages or debug output. The risk is that an operator or automated workflow error could leak AppRole credentials. - name: Mint agent-core-coordinator Vault bootstrap if: ${{ !inputs.dry_run && inputs.playbook == 'engineering-loop' }} run: | set -euo pipefail coordinator_version="$(sed -nE 's/^agent_core_coordinator_version:[[:space:]]*["'"']?([0-9a-f]{40})["'"']?[[:space:]]*$/\1/p' ansible/inventory/host_vars/loop.yml | head -1)" if [ -z "$coordinator_version" ]; then echo "agent-core-coordinator is not pinned to an exact SHA; skipping its dark scaffold bootstrap" exit 0 fi if [ -n "${VAULT_AGENT_CORE_COORDINATOR_ROLE_ID:-}" ] && { [ -n "${VAULT_AGENT_CORE_COORDINATOR_WRAPPED_SECRET_ID:-}" ] || [ -n "${VAULT_AGENT_CORE_COORDINATOR_SECRET_ID:-}" ]; }; then echo "agent-core-coordinator Vault bootstrap already provided by environment" exit 0 fi token_file="/run/vault-agent/github-runner.token" if [ ! -r "$token_file" ]; then echo "::error::${token_file} is not readable by the runner user; re-apply playbooks/ci.yml to refresh Vault Agent token sink permissions" exit 1 fi vault_token="$(cat "$token_file")" export VAULT_ADDR="${VAULT_ADDR:-http://[2a0c:b641:b50:2::c0]:8200}" export VAULT_TOKEN="$vault_token" role_id="$(vault read -field=role_id auth/approle/role/agent-core-coordinator/role-id)" wrapped_secret_id="$(vault write -wrap-ttl=10m -field=wrapping_token -f auth/approle/role/agent-core-coordinator/secret-id)" if [ -z "$role_id" ] || [ -z "$wrapped_secret_id" ]; then echo "::error::failed to mint agent-core-coordinator Vault bootstrap credentials" exit 1 fi echo "::add-mask::$role_id" echo "::add-mask::$wrapped_secret_id" printf 'VAULT_AGENT_CORE_COORDINATOR_ROLE_ID=%s\n' "$role_id" >> "$GITHUB_ENV" printf 'VAULT_AGENT_CORE_COORDINATOR_WRAPPED_SECRET_ID=%s\n' "$wrapped_secret_id" >> "$GITHUB_ENV" { echo "## agent-core-coordinator Vault bootstrap" echo echo "Minted response-wrapped AppRole SecretID for this apply run." } >> "$GITHUB_STEP_SUMMARY" - name: Mint agentic-observatory Vault bootstrap if: ${{ !inputs.dry_run && inputs.playbook == 'engineering-loop' }} run: | set -euo pipefail if [ -n "${VAULT_AGENTIC_OBSERVATORY_ROLE_ID:-}" ] && { [ -n "${VAULT_AGENTIC_OBSERVATORY_WRAPPED_SECRET_ID:-}" ] || [ -n "${VAULT_AGENTIC_OBSERVATORY_SECRET_ID:-}" ]; }; then echo "agentic-observatory Vault bootstrap already provided by environment" exit 0 fi token_file="/run/vault-agent/github-runner.token" if [ ! -r "$token_file" ]; then echo "::error::${token_file} is not readable by the runner user; re-apply playbooks/ci.yml to refresh Vault Agent token sink permissions" exit 1 fi vault_token="$(cat "$token_file")" export VAULT_ADDR="${VAULT_ADDR:-http://[2a0c:b641:b50:2::c0]:8200}" export VAULT_TOKEN="$vault_token" role_id="$(vault read -field=role_id auth/approle/role/agentic-observatory/role-id)" wrapped_secret_id="$(vault write -wrap-ttl=10m -field=wrapping_token -f auth/approle/role/agentic-observatory/secret-id)" if [ -z "$role_id" ] || [ -z "$wrapped_secret_id" ]; then echo "::error::failed to mint agentic-observatory Vault bootstrap credentials" exit 1 fi echo "::add-mask::$role_id" echo "::add-mask::$wrapped_secret_id" printf 'VAULT_AGENTIC_OBSERVATORY_ROLE_ID=%s\n' "$role_id" >> "$GITHUB_ENV" printf 'VAULT_AGENTIC_OBSERVATORY_WRAPPED_SECRET_ID=%s\n' "$wrapped_secret_id" >> "$GITHUB_ENV" { echo "## agentic-observatory Vault bootstrap" echo echo "Minted response-wrapped AppRole SecretID for this apply run." } >> "$GITHUB_STEP_SUMMARY" - name: Mint SOC Agent Vault bootstrap if: ${{ !inputs.dry_run && inputs.playbook == 'soc' }} run: | set -euo pipefail if [ -n "${VAULT_SOC_AGENT_ROLE_ID:-}" ] && { [ -n "${VAULT_SOC_AGENT_WRAPPED_SECRET_ID:-}" ] || [ -n "${VAULT_SOC_AGENT_SECRET_ID:-}" ]; }; then echo "SOC Agent Vault bootstrap already provided by environment" exit 0 fi token_file="/run/vault-agent/github-runner.token" if [ ! -r "$token_file" ]; then echo "::error::${token_file} is not readable by the runner user; re-apply playbooks/ci.yml to refresh Vault Agent token sink permissions" exit 1 fi vault_token="$(cat "$token_file")" export VAULT_ADDR="${VAULT_ADDR:-http://[2a0c:b641:b50:2::c0]:8200}" export VAULT_TOKEN="$vault_token" role_id="$(vault read -field=role_id auth/approle/role/soc-agent/role-id)" wrapped_secret_id="$(vault write -wrap-ttl=10m -field=wrapping_token -f auth/approle/role/soc-agent/secret-id)" if [ -z "$role_id" ] || [ -z "$wrapped_secret_id" ]; then echo "::error::failed to mint SOC Agent Vault bootstrap credentials" exit 1 fi echo "::add-mask::$role_id" echo "::add-mask::$wrapped_secret_id" printf 'VAULT_SOC_AGENT_ROLE_ID=%s\n' "$role_id" >> "$GITHUB_ENV" printf 'VAULT_SOC_AGENT_WRAPPED_SECRET_ID=%s\n' "$wrapped_secret_id" >> "$GITHUB_ENV" { echo "## SOC Agent Vault bootstrap" echo echo "Minted response-wrapped AppRole SecretID for this apply run." } >> "$GITHUB_STEP_SUMMARY" Unpinned Origin/Main Checkout On line 260, when SOC_AGENT_SHA is set, the script runs git rev-parse origin/main and uses that SHA as the network-operations pin. This resolves origin/main at the moment the workflow runs, not at the time of app PR merge. If another promotion PR merges to main between the app PR merge and this workflow execution, the network-operations pin will be set to a later commit than the one reviewed. This creates a gap in the audit trail and could deploy an unreviewed network-operations revision alongside the SOC agent. Password Written to SQL File The PostgreSQL password from Vault is written to a temporary SQL file on disk (line 67) with chmod 0600 before being cleaned up via a trap. Between the write and the trap execution, if the play is interrupted (e.g., by SSH disconnect, timeout, or manual Ctrl+C), the SQL file remains on disk world-readable (root-owned but still on-disk plaintext). On a production coordinator VM, an attacker with brief local access could recover the password. Password material should be passed via stdin or a pipe to psql, never written to a file even with cleanup. sql_file="$(mktemp /tmp/agent-core-coordinator-pg.XXXXXX.sql)" trap 'rm -f "$sql_file"' EXIT chmod 0600 "$sql_file" python3 - <<'PY' > "$sql_file" import os password = os.environ["HYRULE_COORDINATOR_DB_PASSWORD"].replace("'", "''") print("""DO $$ BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '{{ agent_core_coordinator_database_role }}') THEN CREATE ROLE {{ agent_core_coordinator_database_role }} LOGIN PASSWORD '%s'; ELSE ALTER ROLE {{ agent_core_coordinator_database_role }} WITH LOGIN PASSWORD '%s'; END IF; END $$;""" % (password, password)) PY chown postgres:postgres "$sql_file" runuser -u postgres -- psql -v ON_ERROR_STOP=1 -f "$sql_file"

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Accept uppercase hex in SOC readiness check The regex [0-9a-f]{40} only matches lowercase hex. For consistency with the find_pin function in promote-app-pins.py that accepts both cases, add A-F to allow uppercase SHA-1 digests. Otherwise, SOC rollout may be blocked if an upstream repo distributes uppercase hashes. .github/workflows/app-promotion-deploy.yml [173-176] soc_ready = all( - re.search(rf"^{key}:\s*[\"']?[0-9a-f]{{40}}[\"']?\s*$", soc_vars, re.MULTILINE) + re.search(rf"^{key}:\s*[\"']?[0-9a-fA-F]{{40}}[\"']?\s*$", soc_vars, re.MULTILINE) for key in ("soc_agent_version", "soc_network_operations_version") ) Suggestion importance[1-10]: 7 __ Why: The suggestion correctly identifies that the regex on lines 173-176 uses [0-9a-f]{40} without allowing uppercase hex (A-F). Since other relevant functions (e.g., find_pin in promote-app-pins.py) accept both cases, this mismatch could block a valid SOC rollout if upstream SHAs contain uppercase letters. This is a valid and moderately important fix for consistency and correctness.	Medium
Possible issue	Make SHA regex case-insensitive Make SHA_RE case-insensitive to match the hex SHA pattern used in find_pin and update_pin, which accept both lowercase and uppercase hex digits. Without this, render_body will produce an incorrect commit link (instead of compare link) when an old SHA contains uppercase letters. scripts/ci/promote-app-pins.py [13] -SHA_RE = re.compile(r"^[0-9a-f]{40}$") +SHA_RE = re.compile(r"^[0-9a-fA-F]{40}$") Suggestion importance[1-10]: 6 __ Why: The suggestion is accurate: the regex SHA_RE on line 13 of the new hunk is defined as ^[0-9a-f]{40}$, which only matches lowercase hex. The find_pin and update_pin functions (lines 184-189 and 211-214) use A-F in their pattern, so SHA_RE should also be case-insensitive for consistency. This prevents potential display bugs in render_body (line 242). This is a valid correctness improvement of moderate importance.	Low

[github-actions]

Persistent review updated to latest commit 2026c7c

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2026c7c1d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c95e1e96d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Gate the SOC scrape target until node_exporter exists

When this dark scaffold merges before the SOC VM has been provisioned and applied, app-promotion-deploy sees configs/mon/prometheus.yml changed and deploys Prometheus immediately even though the SOC play is skipped until soc_agent_version is a 40-character SHA. This new node-infra target will have up == 0, which the existing NodeExporterDown critical rule in configs/mon/prometheus-rules/noc-tripwire.yml fires on after 2 minutes. Please add this scrape target only with the SOC promotion/apply path, or otherwise suppress it until SOC node_exporter is installed.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Allow SOC to reach the trace collector

When SOC insights or trace emission are enabled, this declares SOC will send to the loop collector on 8770, but the destination firewall in ansible/inventory/host_vars/loop.yml still allows that port only from loop, NOC, and mon. The SOC promotion flow even applies the loop firewall before SOC, so the SOC → agent-core collector path described in the runbook will be dropped and the InsightDecisionRecords needed for SOC promotion criteria cannot populate. Please add peers.soc.ipv6 to the loop 8770 allowlist or keep this egress disabled until the destination rule exists.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Require coordinator approval before probe timers

When a later promotion sets soc_mode to probe_dry/probe_live and enables soc_probe_timer_enabled but forgets soc_coordinator_enabled, this assertion still passes and the role will start soc-probes.timer with SOC_REDTEAM_ALLOW_ACTIVE_PROBES=1 while SOC_COORDINATOR_ENABLED=0. The new probe service is a coordinator-polling worker and the runbook says probe rungs require central exact-scope senior approval, so the guard should require the coordinator to be enabled before any probe timer/probe rung can run.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Restart the NOC coordinator worker on app upgrades

When noc_coordinator_worker_enabled is true and a later promotion changes noc_agent_version, the clone and uv sync tasks only notify restart noc-agent and restart noc-agent-bot; this new state task uses started, so an already-running noc-coordinator-worker.service keeps executing the old checkout until a manual restart or unrelated env render happens. Please include restart noc coordinator worker in the code/dependency-change notifications or otherwise force a restart when the pinned NOC app changes.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Restart the Knowledge coordinator on app upgrades

When knowledge_loop_coordinator_enabled is true and knowledge_loop_version changes, the checkout and dependency sync tasks notify only the Knowledge Loop timer, while this new coordinator service state remains started and does not restart an active hyrule-knowledge-coordinator.service. The long-running coordinator therefore keeps the old code after a promotion until a manual restart or template change occurs; please notify restart knowledge coordinator on runtime checkout/dependency changes too.

Useful? React with 👍 / 👎.