Repository navigation
docs: code-grounded review of maturity.md — finds 4 under-claims, 9 missing rows - #370
Merged
Merged
Conversation
…issing rows
Detailed code-path audit of every claim in docs/maturity.md against the
actual symbol use in controller/src/, inference-router/src/, cli/src/.
Findings:
- A. Under-claims (rows that should change status):
A.1 InferencePolicy daily/monthly token budgets: marked reconciler-only,
actually wired in routes/inference.rs, routes/chat_completions.rs,
routes/anthropic_messages.rs (call to check_budget with policy values)
A.2 kars attest sign/verify: marked roadmap, actually 618 LOC of working
code (canonical-JSON spec hash + SSA field owners + policy refs +
reconcile trace + baseline diff with proper exit codes 0/2/3)
A.3 Signed-OCI allowlist authority flip: marked roadmap, actually 4
branches in controller/src/policy_fetcher.rs:917-931, all wired
A.4 A2A AgentCard JWS verify: marked library-only, accurate for the
AgentCard verifier but the AP2 IntentMandate/PaymentAttempt path IS
wired (routes/a2a.rs:51 imports handle_message_send_with_ap2)
- B. Missing rows (real runtime enforcement not surfaced anywhere):
B.1 7 ValidatingAdmissionPolicies (sandbox-posture-lock, pod-exec-ban,
no-public-router-exposure, null-provider, content-safety-floor,
dev-only-label-immutable, seccomp-auto-stamp)
B.2 MCP server signed-key + JWKS discovery (MCP_JWKS_DIR registry)
B.3 AP2 mandate signing + ledger (replay/window enforcement)
B.4 ToolPolicy commerce/approval/rateLimit blocks
B.5 EgressApproval merging logic (rows exists but lacks digest-change note)
B.6 SignerPolicy ConfigMap hot-reload
B.7 A2AAgent FederationPeer trust circle
B.8 KarsMemory store binding + scope projection
B.9 ContentSafetyFloor per-category severity caps
- C. Accurate rows: 18 confirmed by tracing the symbol through to a route
handler or runtime call site.
Proposed updated maturity table sketched in section 3 (new sections:
Admission-time enforcement, MCP gateway, AP2; expand AGT/ToolPolicy).
Row count would grow from ~32 to ~46.
No fixes applied to docs/maturity.md itself — user asked for the analysis,
fixes are for a separate doc PR.
File: docs/internal/security-validations/2026-05-31-maturity-doc-vs-code.md
(18197 chars).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Pal Lakatos-Toth <pallakatos@github.com>
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
This was referenced May 31, 2026
Pal Lakatos-Toth (pallakatos)
added a commit
that referenced
this pull request
May 31, 2026
… of docs/ (#371) Companion to 2026-05-31-maturity-doc-vs-code.md (PR #370). Same methodology, broader surface: every .md file under docs/. Findings (8 files affected): - docs/roadmap.md: 4 entries already shipped or partially shipped - docs/security.md: 'not yet enforced' callout has 2 stale entries - docs/compliance.md: inherits maturity.md under-claims - docs/api/crd-reference.md: meshAuthBackend 'scaffolded' is stale (EntraAgentIdentity wired end-to-end, verified live on AKS) - docs/architecture.md: macOS Docker Desktop caveat missing, entra-auth-sidecar absent from architecture diagrams - docs/use-cases.md: TUI undersold (1042 LOC, security panel, etc.) - docs/use-cases/exec-brief-walkthrough.md: harness claim outdated (docker now 9/9 as of PR #367) - docs/maturity.md: see PR #370 Total under-claim or outdated entries identified: 8 status changes + 10 missing rows/sections + 1 missing CRD reference page. Three suggested remediation PRs sketched in §H: 1. 'docs: refresh what ships today' (~150 LOC, low risk) 2. 'docs: architecture diagrams + KarsAuthConfig CRD ref' (~300 LOC) 3. 'docs: maturity.md restructure' (~250 LOC, ties into PR #370) No fixes applied — analysis only, consistent with the validate-first, fix-later pattern. Signed-off-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pal Lakatos-Toth (pallakatos)
added a commit
that referenced
this pull request
May 31, 2026
Following PRs #370 (maturity.md review) and #371 (doc-wide audit), this extends the analysis specifically into the architecture/CRD/AGT-boundary doc surface. Methodology: trace every claim about a CRD field, component, deployment, provider variant, or guarantee to the actual code. Findings: A. CRDs (docs/api/crd-reference.md) - KarsSandbox spec table covers 8 fields; code has 17. 9 missing (agent, azureServices, resources, upstreamCompatibility, meshAuth, sigsAgentSandbox, upstreamSandboxRef, aiConformanceReference, customSecurityAttributes) — mostly Phase-6 Entra additions. - KarsSandbox.status missing agentIdentity (operators look this up via kubectl get karssandbox -o jsonpath='{.status.agentIdentity.appId}'). - EntraAgentIdentity status caveats (line 684,694) stale (covered in PR #371 §D.1). B. Architecture (docs/architecture.md) - §Components table lists 4 components; code has 6 first-class (missing mesh-plugin, conformance-runner, eval-corpus). - 'The nine CRDs' subheader should be 'nine workload CRDs' (text body correctly says 11 total). C. AGT boundary (docs/architecture/agt-boundary.md) — MOST OUTDATED: - §1 claims 'A2A AgentCard signing — until AGT ships'; but kars-a2a-core/src/card_signing.rs is 494 LOC of working Ed25519 JWS signer (sign_card + verify_card + TrustedKeys). - §2 provider variant 'AgtRustSdk' doesn't exist in code (it's 'Agt'). - §3 'KarsSandbox.spec.agt.outageMode' field doesn't exist. - §4 'kars never builds a standalone audit chain' — but inference-router/src/audit/merkle.rs exists (library-only Merkle anchoring). Needs footnote. - §5 'kars always builds' list omits 7 VAPs, AP2 trust/ledger, Phase 6 Entra Agent ID surface. D. Architecture diagrams (docs/architecture-diagrams.md) - §7 CRD diagram shows 9 workload CRDs; doc says 'the nine CRDs' but cluster has 11 (missing KarsAuthConfig + KarsPairing). - §8 cluster topology mermaid omits entra-auth-sidecar despite being live in every AKS cluster post-PR #360. - §8 shows a2a-gateway in kars-system but it's not deployed by default (needs gating or removal). - §6 control plane diagram represents controller as single box; code has 11 separate reconciler modules. E. A2A gateway (docs/architecture/a2a-gateway.md) - 8445 listener config-only — accurately disclosed ✅ - Known limitations should drop AP2 if listed (AP2 IS wired). Total: 16 actionable findings; 2 suggested doc-only PRs sketched in §G. No fixes applied — analysis only, per the established validate-first pattern. File: docs/internal/security-validations/2026-05-31-crd-arch-agtboundary-deep-audit.md (21227 chars, 250 lines). Signed-off-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pal Lakatos-Toth (pallakatos)
added a commit
that referenced
this pull request
May 31, 2026
Comprehensive audit of every .md file under docs/ (excluding internal/ and site/) against actual code paths in controller/, inference-router/, cli/, mesh-plugin/, kars-a2a-core/, a2a-gateway/, sandbox-images/, runtimes/, eval-corpus/, deploy/helm/. Methodology (per file): 1. Locate every claim that can be cross-referenced to code 2. Trace the symbol/file/handler/route/CRD field 3. Confirm by reading the call site (not the comment/docstring) 4. Document mismatches with code citations Tracked progress via session-local SQL doc_audit table to make the audit context-window-safe across 62 files. Coverage: 62 / 62 (100%) — 51 newly audited, 11 covered by PR #370/371/372 Findings (this report only): 4 MEDIUM, 0 HIGH, 0 LOW, 0 TYPO 1. architecture/entra-agent-id/01-runtime-token-flow.md — Phase 5b migration not crossreferenced; ASCII diagram shows per-pod sidecar 2. operations/image-versioning.md — claims 'eight container images' including 'five runtime adapter images'; actually 13+ images with 7 runtime adapters 3. cli-reference.md — 'kars headlamp' command not documented despite being wired in cli/src/cli.ts:73 4. upstream-alignment.md — 3 references to cli/src/plugin.ts (file deleted, plugin moved to runtimes/openclaw/src/index.ts) plus stale entrypoint.sh line number (223 vs actual 784) Combined with the 26+ findings in PR #370/371/372, the overall doc-audit total is ~30 findings across docs/ — mostly stale row-status labels in maturity/security/roadmap, a handful of dead file links, one mis-counted image inventory, no factually-wrong security guarantees. Verdict: doc surface is substantially accurate (94% of files have zero findings). Three suggested follow-up doc-only fix PRs sketched in §H. File: docs/internal/security-validations/2026-05-31-full-docs-folder-audit.md (294 lines). Signed-off-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Pal Lakatos-Toth <pallakatos@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
User asked: "can you do a detailed deep analysis on docs/maturity.md - I think we implement more than claimed here - deep investigate the code paths not only comments and doc"
This PR adds a code-grounded review of every claim in
docs/maturity.md. The doc is honestly conservative — but it under-claims in 4 material places and omits 9 capabilities entirely.Headline findings
A. Under-claims (rows that should change status)
kars attest sign/verifyB. Missing rows (entirely absent from maturity.md)
MCP_JWKS_DIR)commerce/approval/rateLimitblocksC. Accurate rows (verified by tracing symbols to call sites)
18 ✅ claims confirmed correct.
Methodology
For each ✅ claim: greped the field name through to a route handler. For 🔵 → ✅ promotions: confirmed the symbol is called outside
#[cfg(test)]. For ⚪ → ✅ promotions: confirmed real CLI/binary output exists.Outcome
Proposed updated maturity table sketched in §3 of the report. Row count would grow from ~32 to ~46. No edits to
docs/maturity.mdin this PR — that's intentional per the user's instruction to validate first, fix later in a separate PR.File added
docs/internal/security-validations/2026-05-31-maturity-doc-vs-code.md(244 lines)