phase2/mcp-reconciler — full McpServer reconciler + JWKS + /mcp mount (S1)

[Pal Lakatos-Toth (pallakatos)]

Phase 2 slice S1 — phase2/mcp-reconciler

Closes §14.6 column 3 (MCP 2026 server CRD): schema → full reconciler + route mount.

Phase 1 shipped the McpServer CRD schema only. This slice ships the full reconciler, helm CRD, and the /mcp mount in the inference-router. Establishes the JWKS Secret + ConfigMap + route-mount pattern that S2 (ToolPolicy) and S3 (A2AAgent) reuse.

What's new

• controller/src/mcp_server_reconciler.rs — full reconciler (~600 LOC modeled on pairing_reconciler.rs):
  • Finalizer azureclaw.azure.com/mcpserver-cleanup (cascades Secret + ConfigMap deletion).
  • Ed25519 keypair generated via rand::rng() + fill_bytes + SigningKey::from_bytes (mirrors mesh_peer/mod.rs::MeshIdentity::generate — avoids enabling pkcs8 feature on ed25519-dalek 2.2.0). Raw 32-byte private + 32-byte public stored in Secret of type azureclaw.azure.com/mcp-signing-key with a kid annotation.
  • JWKS fetched via OpenID Discovery (/.well-known/openid-configuration → jwks_uri), https-only, 10s timeout. Cached as ConfigMap mcp-{name}-jwks. Pluggable JwksFetcher trait keeps tests network-free.
  • SSA via Patch::Apply, field manager azureclaw-controller/mcp (per §10.4 Bump jsonwebtoken from 9.3.1 to 10.3.0 #1).
  • Conditions (Ready / Progressing / Degraded) reused from status/conditions.rs.
• controller/src/mcp_server.rs — added signing_key_ref and jwks_config_map_ref LocalObjectRef fields on McpServerStatus.
• controller/src/helm_drift.rs — drift detector. Unit test parses deploy/helm/azureclaw/templates/crd-mcpserver.yaml, compares (canonicalized) with mcp_server_crd(). One-shot DUMP_MCP_CRD_YAML=1 test regenerates the helm template on intentional schema changes.
• deploy/helm/azureclaw/templates/crd-mcpserver.yaml — auto-generated helm-side CRD mirror.
• inference-router/src/main.rs::build_mcp_router() — selects between dev mcp_route() and OAuth-2.1-gated protected_mcp_route() based on env vars (MCP_PRODUCTION_MODE, MCP_JWKS_PATH, MCP_OAUTH_AUDIENCE, MCP_OAUTH_ISSUER, MCP_OAUTH_REQUIRED_SCOPES). On a misconfigured production mode the router refuses to mount rather than silently falling back to unauthenticated.
• inference-router/src/mcp/oauth.rs::OAuthVerifierConfig::from_jwks_file — new constructor for controller-mounted JWKS.

Audit

• docs/security-audits/2026-04-27-phase2-mcp-reconciler.md — full audit:
  • §0 Existing implementation surveyed — enumerates the 17 Phase 0/1 seams reused (per the no-duplication rule added to the Phase 2 plan).
  • §1 Threat model delta, §2 OWASP MCP Top 10 (MCP-01/04/08), §3 Auth/authz path, §4 Key custody, §5 Egress surface, §6 Audit events, §7 Failure modes, §8 Negative-test coverage, §9 Out-of-scope, §10 Verification table.
  • Two sign-offs at the bottom (Copilot + Pal Lakatos-Toth).

Tests

Suite	Result
Controller bins (cargo test -p azureclaw-controller --bins)	74 → 162 pass (+9 in mcp_server_reconciler::tests, +2 in helm_drift::tests, rest are dormant Phase 1 tests now compiled)
Workspace (cargo test --workspace)	829 pass, 0 fail
cargo fmt --all -- --check	✅
cargo clippy --all-targets -- -D warnings	✅
ci/no-stubs.sh	✅
ci/no-custom-crypto.sh	✅
ci/check-loc.sh	✅
ci/security-audit-required.sh	✅
ci/no-null-provider-prod.sh	✅
ci/a2a-module-isolation.sh	✅
ci/vendored-patch-audit.sh	✅
Helm CRD drift test	✅
CLI typecheck + lint (sanity, no CLI changes)	✅

§0.3 success-gate self-check

• No black-box compat regression
• All conformance-corpus negative tests pass
• No file grew past Phase 2 cap; touched files shrank or held
• Audit doc with two sign-offs merged in same PR
• Zero TODO / unimplemented! / panic! / .stub on production paths
• Zero custom-crypto lint violations
• Docs updated (CHANGELOG, security-audits/, schema doc on mcp_server.rs)
• Unit + negative-test coverage for the new reconciler + JWKS fetcher
• No duplication — every reused seam called out in audit doc §0; no parallel JWKS verifier, no parallel SSA helper, no parallel reconcile loop

What's deferred (per audit doc §9)

• Per-sandbox JWKS ConfigMap mount path (controller patching the router pod's deployment volumes) — handled in S7 (phase2-conditions-ssa-leader) which touches every reconciler.
• MCP_PROTOCOL_VERSION header plumbing into build_mcp_router() — couples to EchoDispatcher real-tool semantics, out of scope for S1.
• Conformance corpus it.todo placeholders — replaced incrementally by S2/S3.

Phase 2 plan reminder

This is slice 1 of ~17. Same operational pattern as Phase 1: small phase2/<slice> PRs into dev, then a final phase2-integration PR dev → main (S18). See docs/implementation-plan.md §8 + docs/competitive.md §14.6 + §15.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com