Repository navigation
feat: add TypeScript directive observability tooling - #16
Merged
Merged
Conversation
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThis PR migrates the agent-directives evaluation framework from Bash and Python to TypeScript npm scripts. It adds project configuration ( ChangesTypeScript Eval Tooling Migration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
AGENTS.md (1)
9-9:⚠️ Potential issue | 🟡 Minor | ⚡ Quick win"No package manager" claim is stale after adding npm tooling.
Line 9 states "there is no package manager, build artifact, or runtime application" but the PR introduces
package.jsonand requiresnpm install. Suggest updating to acknowledge that npm is used for developer scripts only, e.g.:-- Content-first Markdown repository; there is no package manager, build artifact, or runtime application. +- Content-first Markdown repository; the directives and templates work without any build step. npm is used only for developer tooling scripts (type-checking, validation, eval reporting) — not for runtime or distribution.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@AGENTS.md` at line 9, Update the sentence claiming there is "no package manager, build artifact, or runtime application" to acknowledge that npm tooling has been added for developer scripts; edit the text in AGENTS.md so it states that npm (package.json) is present only for developer scripts and local tooling (no runtime or build artifact), e.g., replace or append to the existing line to clearly indicate npm is used for developer-only tasks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@package.json`:
- Line 14: Update the dev dependency version for the TypeScript Node typings:
replace the "@types/node": "^20.12.12" entry with a supported LTS range (e.g.
"@types/node": "^24" or "@types/node": "^22") so the project targets a non-EOL
Node release; edit the package.json dependency line for "@types/node"
accordingly and run your package manager to refresh lockfile/install.
In `@scripts/eval-scenario.ts`:
- Around line 144-148: The code currently calls spawnSync('claude', ...) and
only records result.status, which will be null when the binary or cwd is missing
and result.error is never inspected; update the block around spawnSync to check
result.error first (inspect result.error.code, e.g., 'ENOENT') and when present
write a clear diagnostic to stderr (console.error) and record a non-null exit
indicator in the manifest (set manifest.exit_status to a distinct value like 127
or include manifest.error_message with result.error.message) before writing
manifestPath via writeFileSync and calling process.exit with a concrete status;
reference the existing variables/result object (spawnSync call, result,
manifest, manifestPath, writeFileSync, process.exit) to locate and implement
this check and logging.
In `@scripts/validate-directives.ts`:
- Around line 78-80: The current build of the skills list unconditionally
appends "SKILL.md" for every subdirectory (variable skills) so
validateFrontmatter (which calls read()/readFileSync) can throw and crash when a
SKILL.md is missing; update the code that constructs skills to check for the
existence of the SKILL.md file (e.g., fs.existsSync or stat) before including
the path, and if missing call the validator's fail() with a structured error
mentioning the directory name and missing SKILL.md (not letting
read()/readFileSync throw). Locate references to skills, validateFrontmatter,
read()/readFileSync and fail() to implement this change.
---
Outside diff comments:
In `@AGENTS.md`:
- Line 9: Update the sentence claiming there is "no package manager, build
artifact, or runtime application" to acknowledge that npm tooling has been added
for developer scripts; edit the text in AGENTS.md so it states that npm
(package.json) is present only for developer scripts and local tooling (no
runtime or build artifact), e.g., replace or append to the existing line to
clearly indicate npm is used for developer-only tasks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 6c604e04-7b87-468c-bfe1-4dd206d93621
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (14)
.gitignoreAGENTS.mdREADME.mddirectives/adaptive-routing.mdevals/README.mdevals/report-results.pyevals/report-results.tsevals/results/README.mdevals/run-scenario.shevals/scenarios/directive-load-logging.mdpackage.jsonscripts/eval-scenario.tsscripts/validate-directives.tstsconfig.json
💤 Files with no reviewable changes (1)
- evals/report-results.py
Collaborator
Author
|
Addressed the valid review comments in a43fac5:
Verification:
|
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/eval-scenario.ts`:
- Around line 47-48: The code uses the variable scenario to build filesystem
paths (e.g., scenarioFile) without validation allowing path traversal; before
constructing paths or directory names (and at other uses around the other
occurrences), validate and sanitize scenario: ensure it's non-empty and matches
a safe whitelist (e.g., /^[A-Za-z0-9-_]+$/) or replace it with
path.basename(scenario) to strip any directory components, then reject or error
on invalid input; apply the same validation/sanitization wherever scenario is
used to build paths (including the other mentioned occurrences).
- Line 99: Protect the spawnSync('git', ['rev-parse', '--short', 'HEAD'], { cwd:
repoRoot, encoding: 'utf8' }) call used to set commit by checking the returned
result object before accessing stdout.trim(); if the result has an error, a
non-zero status, or stdout is falsy, use a safe fallback (e.g. 'unknown' or ''),
optionally log the git error/result for diagnostics, and ensure the rest of the
script continues to write the manifest and run diagnostics using that fallback
value instead of throwing in the commit assignment.
- Line 8: Replace the URL.pathname string manipulation used to compute repoRoot
with Node's fileURLToPath conversion to correctly decode percent-encoded
characters and handle platform-specific paths: import fileURLToPath from 'url'
and compute repoRoot by passing new URL('..', import.meta.url) into
fileURLToPath instead of using .pathname; make identical changes where the same
pattern occurs in the other files (the occurrences referenced in this review:
scripts/validate-directives.ts and evals/report-results.ts) so all repo-root
path calculations use fileURLToPath.
In `@scripts/validate-directives.ts`:
- Around line 97-100: The code calls read('directives/adaptive-routing.md')
without guarding for a missing file which can throw; wrap the read call in a
safe check (e.g., try/catch or pre-check with fs.existsSync) and if the file is
missing set adaptive to an empty string or emit a structured warning via
warn(...) so validation continues; update the block around the read(...)
assignment for the variable adaptive and keep the subsequent loop for (const
skill of skills) unchanged so it will report missing mentions rather than
crashing.
- Line 5: The repoRoot calculation uses URL.pathname which is platform-brittle;
replace it with Node's fileURLToPath to get a canonical filesystem path. Import
fileURLToPath from 'url' and change the expression that defines repoRoot
(currently using new URL('..', import.meta.url).pathname.replace(/\/$/, '')) to
use fileURLToPath(new URL('..', import.meta.url)) and remove the manual
trailing-slash regex logic so repoRoot is normalized across OSes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 7b829bad-9482-4419-a514-2906ae4eff43
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (3)
package.jsonscripts/eval-scenario.tsscripts/validate-directives.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
Collaborator
Author
|
Addressed the valid comments posted after
Verification:
|
This was referenced May 9, 2026
This was referenced May 26, 2026
8 tasks
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
npm run checknpm run eval:scenario -- --print-only directive-load-loggingsmoke-tested manifest creationnpm run eval:reportbash -n evals/run-scenario.shgit diff --checkSummary by CodeRabbit
New Features
Documentation
Chores