Skip to content

test: cover missing schema path diagnostics - #698

Merged
flyingrobots merged 2 commits into
mainfrom
test/cli-missing-schema-path
Jul 5, 2026
Merged

flyingrobots merged 2 commits into
mainfrom
test/cli-missing-schema-path

Conversation

@flyingrobots

Copy link
Copy Markdown
Owner

Summary

  • Add a CLI integration regression for wesley schema lower --schema <missing>
  • Assert nonzero exit, empty stdout, and a clear missing schema diagnostic
  • Record the diagnostic coverage in the changelog

Closes #691

Validation

  • cargo test -p wesley-cli --test cli schema_lower_reports_missing_schema_path
  • cargo test -p wesley-cli --test cli schema
  • pnpm exec prettier --check CHANGELOG.md
  • git diff --check
  • pnpm run preflight
  • pre-push Rust product preflight

@coderabbitai

coderabbitai Bot commented Jul 5, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@flyingrobots, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 27 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: f4d5d577-57a0-4abf-a788-98b9fd10f13b

📥 Commits

Reviewing files that changed from the base of the PR and between dc5543d and bae6cf7.

📒 Files selected for processing (2)
  • CHANGELOG.md
  • crates/wesley-cli/tests/cli.rs
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch test/cli-missing-schema-path

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown

🔍 The Case of Pull Request #698

Schema Sets

  • ecommerce
  • reference

Schema Set ecommerce

Plain-English Readout

  • Holmes (evidence investigation): The Holmes report is unavailable because the workflow finished without a readable holmes-report.json artifact.
  • Watson (independent verification): The Watson report is unavailable because the workflow finished without a readable watson-report.json artifact.
  • Moriarty (trend forecast): The Moriarty forecast is unavailable because the workflow finished without a readable moriarty-report.json artifact.

Suggested next actions

  1. Regenerate the HOLMES artifacts and make sure holmes-report.json is uploaded before trusting this PR summary.
  2. Regenerate the WATSON artifacts and make sure watson-report.json is uploaded before trusting this PR summary.
  3. Regenerate the MORIARTY artifacts and make sure moriarty-report.json is uploaded before trusting this PR summary.
🕵️ SHA-lock HOLMES full report for ecommerce (click to expand)

Report unavailable for holmes: readable holmes-report.md artifact not found.

🩺 Dr. WATSON full report for ecommerce (click to expand)

Report unavailable for watson: readable watson-report.md artifact not found.

🔮 Professor MORIARTY full report for ecommerce (click to expand)

Report unavailable for moriarty: readable moriarty-report.md artifact not found.


Schema Set reference

Plain-English Readout

  • Holmes (evidence investigation): The Holmes report is unavailable because the workflow finished without a readable holmes-report.json artifact.
  • Watson (independent verification): The Watson report is unavailable because the workflow finished without a readable watson-report.json artifact.
  • Moriarty (trend forecast): The Moriarty forecast is unavailable because the workflow finished without a readable moriarty-report.json artifact.

Suggested next actions

  1. Regenerate the HOLMES artifacts and make sure holmes-report.json is uploaded before trusting this PR summary.
  2. Regenerate the WATSON artifacts and make sure watson-report.json is uploaded before trusting this PR summary.
  3. Regenerate the MORIARTY artifacts and make sure moriarty-report.json is uploaded before trusting this PR summary.
🕵️ SHA-lock HOLMES full report for reference (click to expand)

Report unavailable for holmes: readable holmes-report.md artifact not found.

🩺 Dr. WATSON full report for reference (click to expand)

Report unavailable for watson: readable watson-report.md artifact not found.

🔮 Professor MORIARTY full report for reference (click to expand)

Report unavailable for moriarty: readable moriarty-report.md artifact not found.


📚 Glossary (what the Holmes terms mean)
  • HOLMES: Wesley’s main evidence investigation. It decides whether the cited proof is strong enough to justify shipping this commit.
  • WATSON: An independent verification pass. It checks Holmes’s citations and score math instead of trusting them blindly.
  • MORIARTY: A readiness forecast over time. It is advisory trend analysis, not the release gate itself.
  • Schema coverage score (SCS): How much of the schema has direct supporting evidence across generated artifacts and cited proof.
  • Test confidence index (TCI): How much test evidence exists for constraints, policies, relationships, and operations.
  • Migration risk index (MRI): How risky the schema change is to roll out. Lower is better.
  • Evidence trust: Whether the report is backed by exact citations, whole-file citations, or coarse references. Weak trust means the claim may be directionally right but not specific enough to trust blindly.
  • Citation quality: A count of exact line-span citations versus whole-file or coarse references.
  • ELEMENTARY: Ready to ship based on the current evidence.
  • REQUIRES INVESTIGATION: More work or review is needed before shipping.
  • YOU SHALL NOT PASS: Do not ship this change in its current state.

Machine-readable reports are grouped by schema set in workflow artifacts.


Filed at 221B Repository Street

@flyingrobots

Copy link
Copy Markdown
Owner Author

Activity Summary

Item Severity File Commit Outcome
Missing schema path CLI diagnostic coverage P3 crates/wesley-cli/tests/cli.rs, CHANGELOG.md 2d7b7cb9 Clean. Added regression for schema lower --schema <missing>; PR body includes Closes #691.
Main merge after #697 P4 CHANGELOG.md bae6cf70 Clean. Regular merge from main; conflict resolved by preserving both changelog entries.

Verification:

  • cargo test -p wesley-cli --test cli schema_lower_reports_missing_schema_path
  • cargo test -p wesley-cli --test cli schema
  • pnpm exec prettier --check CHANGELOG.md
  • git diff --check
  • pnpm run preflight
  • pre-push Rust product preflight
  • gh pr checks 698

Review state: no unresolved review threads; no Changes Requested; CodeRabbit required status is passing with only a visible rate-limit notice. HOLMES-family jobs pass.

@flyingrobots
flyingrobots merged commit 2cc9eed into main Jul 5, 2026
20 checks passed
@flyingrobots
flyingrobots deleted the test/cli-missing-schema-path branch July 5, 2026 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(cli): cover missing schema path diagnostics

1 participant