Skip to content

docs: publish assurance capability matrix - #687

Merged
flyingrobots merged 21 commits into
docs/plain-wesley-onboardingfrom
docs/assurance-capability-matrix
Jul 5, 2026
Merged

flyingrobots merged 21 commits into
docs/plain-wesley-onboardingfrom
docs/assurance-capability-matrix

Conversation

@flyingrobots

@flyingrobots flyingrobots commented Jul 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

Closes #679.

Stacked on #686. This PR keeps the diff scoped to the assurance-positioning issue.

Why

The public assurance story needed a concrete source of truth that separates shipped native CLI commands, retained JavaScript workflows, Rust foundation code, and concept-only vocabulary. Without that split, Wesley's assurance language could imply more executable native capability than the current product surface provides.

Changes

  • Adds docs/reference/assurance-capability-matrix.md with shipped native CLI, shipped JS/transitional, internal foundation, and concept/design-only states.
  • Distinguishes executable Wesley evidence commands from transitional Holmes-family workflows, Rust Holmes foundation code, and BLADE concept vocabulary.
  • Classifies Watson verification and Moriarty prediction as retained JS/transitional surfaces while keeping native CLI parity out of scope.
  • Scopes SHIPME certificates to matching path-filtered main pushes instead of every landed main SHA.
  • Routes README, GUIDE, ARCHITECTURE, END_TO_END, topics, and docs front door to the matrix.

Method Evidence

  • Review-thread fixes are in e5867c65.
  • The Assurance Capability Matrix is the single shipped-status authority.
  • The matrix now reflects .github/workflows/cert-shipme.yml path filters and retained @wesley/holmes Watson/Moriarty executable surfaces.

Tracker Hygiene

Risk

Low. This is documentation-only, but it affects public positioning. The main risk is overstating or understating assurance surfaces; the matrix now scopes each claim to an executable command or workflow.

Backout

Revert this PR or revert e5867c65 if the status taxonomy needs a different release boundary.

Testing

  • pnpm exec prettier --check docs/GUIDE.md docs/reference/assurance-capability-matrix.md docs/topics/assurance-evidence.md docs/topics/plain-wesley.md
  • cargo xtask docs-check
  • git diff --check
  • pnpm run preflight
  • pre-push Rust product preflight

Checklist

  • Documentation updated.
  • Review-thread fixes pushed.
  • Local validation passed.
  • PR remains non-draft.

Summary by CodeRabbit

  • New Features

    • Added a new assurance capability matrix reference and linked it throughout the docs and changelog.
    • Expanded documentation navigation to include the new matrix.
  • Documentation

    • Clarified which assurance-related terms and surfaces are shipped, transitional, internal, or concept-only.
    • Added guidance to avoid overstating what current evidence and workflows prove.
    • Updated the truth manifest to include the new reference page.

@coderabbitai

coderabbitai Bot commented Jul 4, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds a new "Assurance Capability Matrix" reference document distinguishing shipped native CLI, shipped JS/transitional, internal foundation, and concept-only assurance surfaces, then updates README, ARCHITECTURE, GUIDE, END_TO_END, topics docs, docs/README, CHANGELOG, and truth-manifest.json to reference and align with it.

Changes

Assurance Capability Matrix Documentation

Layer / File(s) Summary
New assurance capability matrix reference
docs/reference/assurance-capability-matrix.md, docs/truth-manifest.json, CHANGELOG.md
Adds a new doc defining status states, a capability matrix table, a public claim rule, Rust Holmes MVP deferral notes, and end-to-end evidence examples; registers the doc in the truth manifest and CHANGELOG.
Navigation links to the matrix
docs/README.md, docs/topics/README.md
Adds links to the new matrix in the Signposts table, Product Orientation section, and topic-map Authority references.
README and ARCHITECTURE alignment
README.md, docs/ARCHITECTURE.md
Adds assurance-status guidance in Quick Start, rephrases the "What Wesley Owns" bullet, and clarifies executable assurance status determination.
GUIDE and END_TO_END alignment
docs/GUIDE.md, docs/END_TO_END.md
Updates guide sections and Stage 6 text to reference the matrix and clarify shipped versus conceptual assurance vocabulary, including Watson/Moriarty/BLADE status.
Topics docs alignment
docs/topics/assurance-evidence.md, docs/topics/plain-wesley.md
Expands the Current Surfaces table with state/use-for columns, adds concept-only vocabulary rules, and links the matrix from Related Authority and Advanced Vocabulary Boundary sections.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested labels: enhancement

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The docs add the matrix, reframe assurance claims, defer Rust Holmes MVP, and align the referenced guides with the issue's acceptance criteria.
Out of Scope Changes check ✅ Passed The changes stay focused on assurance positioning docs and navigation updates, with no clear unrelated code or behavior changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check ✅ Passed It is concise and accurately summarizes the new assurance capability matrix docs.
Description check ✅ Passed It covers the main template sections with summary, why, changes, evidence, risk, backout, testing, and checklist.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/assurance-capability-matrix

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/GUIDE.md`:
- Around line 171-172: The shipped-status rule in GUIDE.md should be driven by
the matrix entry, not by whether a page cites a shipped command or workflow.
Update the wording around Watson, Moriarty, and BLADE so it explicitly keys off
the matrix as the single source of truth, and adjust any nearby phrasing to
avoid using page citations as the criterion.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 6ec5a41a-64b2-4cf4-8ccc-7fe9f3f7ec2a

📥 Commits

Reviewing files that changed from the base of the PR and between c576d29 and 45ac7fe.

📒 Files selected for processing (11)
  • CHANGELOG.md
  • README.md
  • docs/ARCHITECTURE.md
  • docs/END_TO_END.md
  • docs/GUIDE.md
  • docs/README.md
  • docs/reference/assurance-capability-matrix.md
  • docs/topics/README.md
  • docs/topics/assurance-evidence.md
  • docs/topics/plain-wesley.md
  • docs/truth-manifest.json

Comment thread docs/GUIDE.md Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45ac7fe212

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread docs/reference/assurance-capability-matrix.md Outdated
Comment thread docs/reference/assurance-capability-matrix.md Outdated
@flyingrobots

Copy link
Copy Markdown
Owner Author

Self-Code Review Findings

@codex please sanity-check these self-discovered documentation consistency findings before merge.

Severity Source File Lines Issue Planned mitigation
P2 Self docs/topics/assurance-evidence.md 36-37 The topic repeats the same shipped-status rule CodeRabbit flagged in docs/GUIDE.md: it keys off page citations rather than the Assurance Capability Matrix. Replace the rule with matrix-owned shipped status and clarify Watson/Moriarty/BLADE status follows the matrix.
P3 Self docs/topics/plain-wesley.md 179-182 The Plain Wesley advanced-vocabulary note says terms are experimental unless a page links to a shipped command, which weakens the matrix as the single source of truth. Point readers to the matrix as the status authority instead of using page links as the criterion.

@github-actions

github-actions Bot commented Jul 5, 2026 •

Copy link
Copy Markdown

🔍 The Case of Pull Request #687

Schema Sets

  • ecommerce
  • reference

Schema Set ecommerce

Plain-English Readout

  • Holmes (evidence investigation): The Holmes report is unavailable because the workflow finished without a readable holmes-report.json artifact.
  • Watson (independent verification): The Watson report is unavailable because the workflow finished without a readable watson-report.json artifact.
  • Moriarty (trend forecast): The Moriarty forecast is unavailable because the workflow finished without a readable moriarty-report.json artifact.

Suggested next actions

  1. Regenerate the HOLMES artifacts and make sure holmes-report.json is uploaded before trusting this PR summary.
  2. Regenerate the WATSON artifacts and make sure watson-report.json is uploaded before trusting this PR summary.
  3. Regenerate the MORIARTY artifacts and make sure moriarty-report.json is uploaded before trusting this PR summary.
🕵️ SHA-lock HOLMES full report for ecommerce (click to expand)

Report unavailable for holmes: readable holmes-report.md artifact not found.

🩺 Dr. WATSON full report for ecommerce (click to expand)

Report unavailable for watson: readable watson-report.md artifact not found.

🔮 Professor MORIARTY full report for ecommerce (click to expand)

Report unavailable for moriarty: readable moriarty-report.md artifact not found.


Schema Set reference

Plain-English Readout

  • Holmes (evidence investigation): The Holmes report is unavailable because the workflow finished without a readable holmes-report.json artifact.
  • Watson (independent verification): The Watson report is unavailable because the workflow finished without a readable watson-report.json artifact.
  • Moriarty (trend forecast): The Moriarty forecast is unavailable because the workflow finished without a readable moriarty-report.json artifact.

Suggested next actions

  1. Regenerate the HOLMES artifacts and make sure holmes-report.json is uploaded before trusting this PR summary.
  2. Regenerate the WATSON artifacts and make sure watson-report.json is uploaded before trusting this PR summary.
  3. Regenerate the MORIARTY artifacts and make sure moriarty-report.json is uploaded before trusting this PR summary.
🕵️ SHA-lock HOLMES full report for reference (click to expand)

Report unavailable for holmes: readable holmes-report.md artifact not found.

🩺 Dr. WATSON full report for reference (click to expand)

Report unavailable for watson: readable watson-report.md artifact not found.

🔮 Professor MORIARTY full report for reference (click to expand)

Report unavailable for moriarty: readable moriarty-report.md artifact not found.


📚 Glossary (what the Holmes terms mean)
  • HOLMES: Wesley’s main evidence investigation. It decides whether the cited proof is strong enough to justify shipping this commit.
  • WATSON: An independent verification pass. It checks Holmes’s citations and score math instead of trusting them blindly.
  • MORIARTY: A readiness forecast over time. It is advisory trend analysis, not the release gate itself.
  • Schema coverage score (SCS): How much of the schema has direct supporting evidence across generated artifacts and cited proof.
  • Test confidence index (TCI): How much test evidence exists for constraints, policies, relationships, and operations.
  • Migration risk index (MRI): How risky the schema change is to roll out. Lower is better.
  • Evidence trust: Whether the report is backed by exact citations, whole-file citations, or coarse references. Weak trust means the claim may be directionally right but not specific enough to trust blindly.
  • Citation quality: A count of exact line-span citations versus whole-file or coarse references.
  • ELEMENTARY: Ready to ship based on the current evidence.
  • REQUIRES INVESTIGATION: More work or review is needed before shipping.
  • YOU SHALL NOT PASS: Do not ship this change in its current state.

Machine-readable reports are grouped by schema set in workflow artifacts.


Filed at 221B Repository Street

@flyingrobots

Copy link
Copy Markdown
Owner Author

Activity Summary

Issue Severity Source File Commit / Action Outcome
Tie shipped status to the matrix, not page citations P2 CodeRabbit docs/GUIDE.md e5867c65 Guide now says Watson/Moriarty/BLADE shipped status is determined by the Assurance Capability Matrix. Thread PRRT_kwDOPoavfs6OUW2n is resolved.
Reclassify Watson and Moriarty as shipped JS surfaces P2 Codex review docs/reference/assurance-capability-matrix.md e5867c65 Matrix now lists Watson verification and Moriarty prediction as shipped JS/transitional surfaces while denying native Rust CLI parity. Resolved PRRT_kwDOPoavfs6OUXjP.
Scope SHIPME artifacts to workflow-matching main pushes P2 Codex review docs/reference/assurance-capability-matrix.md e5867c65 Matrix now scopes SHIPME to matching path-filtered main pushes and states that not every landed SHA receives an artifact. Resolved PRRT_kwDOPoavfs6OUXjR.
Sibling topic repeated page-citation status rule P2 Self docs/topics/assurance-evidence.md e5867c65 Topic now points to the matrix as the status authority.
Plain Wesley inferred shipped status from page links P3 Self docs/topics/plain-wesley.md e5867c65 Advanced-vocabulary note now tells readers not to infer shipped status from page links.
PR description missed template sections P4 CodeRabbit pre-merge check PR metadata PR body edit Added Why, Changes, Method Evidence, Tracker Hygiene, Risk, Backout, Testing, and Checklist sections; CodeRabbit description check now passes.

Validation:

  • RED checks: rg assertions for missing matrix-owned wording, Watson/Moriarty JS rows, and SHIPME path-filter scope failed before the fix.
  • GREEN checks: same rg assertions passed after the fix.
  • pnpm exec prettier --check docs/GUIDE.md docs/reference/assurance-capability-matrix.md docs/topics/assurance-evidence.md docs/topics/plain-wesley.md
  • cargo xtask docs-check
  • git diff --check origin/docs/plain-wesley-onboarding...HEAD
  • pnpm run preflight
  • pre-push Rust product preflight
  • gh pr checks 687 --repo flyingrobots/wesley

@flyingrobots
flyingrobots merged commit b1ad1b3 into docs/plain-wesley-onboarding Jul 5, 2026
16 checks passed
@flyingrobots
flyingrobots deleted the docs/assurance-capability-matrix branch July 5, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant