Repository navigation
fix: validation - RegistryValidator ignores the loaded registry schema #3522
Description
RegistryValidator.validateObject (scripts/validation/lib/registry-validator.js L58–L93) loads registry-schema.json into this.schema but never evaluates it. It passes any object with a truthy agents, skills or entries field, so {"skills":[{}]} is reported valid although it matches none of the schema's oneOf branches.
Raised by CodeRabbit on #3500 (thread on contracts/registry-schema.json L5–L8). #3500 only moves specs, so the code fix is tracked here.
Fix
- Validate with
ajv(already a dependency,^8.17.1) againstthis.schema, keeping theoneOfbranches. The schema uses$defs, so compile with the 2019-09 draft or rename todefinitions. - Keep the
entriesformat only if a producer still emits it; otherwise drop it. - Add tests:
{"skills":[{}]}fails; a valid consolidated registry passes. audit:registryis still a stub and does not call the validator; out of scope.
Verified 2026-09-23: with the #3500 schema compiled in ajv, {"skills":[{}]} fails and a well-formed consolidated registry passes.
Activity
linear-code commentedon Sep 23, 2026
LS-4260
docs(specs): address the second CodeRabbit review on #3500
docs(specs): address the second CodeRabbit review on #3500 (mirrored to
fix(validation): address CodeRabbit review of the registry validator (#…
docs: add AI feedback response for #3500 (#3500)
docs: record the Qodo pass on the current head (#3500)
docs: specs - land CI remediation, changelog agent and agent consolid…
test: cover split registry shapes against the enforced schema
eleshar commentedon Sep 26, 2026
Fixed on #3550 (
e4d38861e2), with review feedback resolved.Changes
RegistryValidatorcompiles the loaded schema withajv(strict: false,allErrors: true) plusajv-formats, and evaluates it invalidateObject. The schema uses$defs, so no rename todefinitionswas needed. When no schema compiles, it falls back to the previous structural check.entriesandgeneratedAtchecks are kept and run on both the schema and fallback paths, since the schema does not declare them.{skills:[{}]}fails; a well-formed consolidated registry passes; a non-object is rejected; a missing schema file falls back.Acceptance criteria verified
Both check out against the current merged schema (the
oneOf/$defsform from #3500), not the older shape the branch was written against.Mutation-checked: disabling the schema evaluation makes 2 of the 8 tests fail, so the coverage genuinely detects the original bug rather than passing incidentally.
Follow-on work
oneOfbranch, so a standalone agents registry (required by spec 014 FR-004) cannot validate.validateAllRegistries()has no caller andaudit:registryis still a stub, so this is not yet breaking anything, but it will be as soon as either is wired up.consolidatedSkillsRegistryandcategorySkillsRegistryare separateoneOfbranches, so skills-only registries validate correctly. Tests now pin that, so a future change cannot silently break it.Also in #3550
Merged
developinto the branch — the base-branch update check had been failing — and resolved theCHANGELOG.mdunion-merge conflict by keeping both sides.fix: fail closed when the registry schema cannot be compiled
eleshar commentedon Sep 26, 2026
Update: a third defect was found during Linear code review and fixed at
95d380740f.Broken schemas silently disabled validation
compileSchema()caught every ajv compilation error and returnednull, whichvalidateObject()could not distinguish from a missing schema file. The result was a silent fall-back to the permissive structural check — the exact behaviour #3522 exists to remove.Reproduced with a schema containing an unresolvable
$ref:Reachable in practice through a typo in a
$ref, or a JSON Schema draft the bundled ajv cannot handle. Qodo raised it and rated the relevance Weak; that rating is wrong, because this path defeats the PR's entire purpose while still reporting success.Fix
compileSchema()recordsthis.schemaError;validateObject()returns invalid for every registry until the schema is corrected. An absent schema keeps its documented structural fallback, so the two failure modes are no longer indistinguishable to callers:Regression test added using a schema ajv cannot compile; mutation-checked — removing the fail-closed branch fails it.
Current state
registry-validator.test.js; full suite 290/290 suites, 5698 passedLeaving this issue open until #3550 merges.
fix: validation - enforce the loaded registry schema (#3522) (#3550)
eleshar commentedon Sep 26, 2026
Merged to
developasfc3cb0a8f6(squash, #3550) and verified on a clean checkout of the merge commit.Behavioural verification on merged develop
Every case run against
RegistryValidatoras it now exists ondevelop:{skills:[{}]}— the original #3522 bugskillskeyBad_ID(pattern violation)type: 'javascript'(not in enum)7/7 as expected. The malformed registry that previously validated as
trueis now correctly rejected.Fail-closed path, added during review:
{skills:[{}]}Regression status
registry-validator.test.js: 9/9fc3cb0a8f6: 0 failures across all check runsLanded
3 files, +241/−2:
CHANGELOG.md(entry present exactly once),registry-validator.js,registry-validator.test.js.The 2 duplicate CHANGELOG entries in the file are the pre-existing ones under
[1.0.0], tracked by #3451 — not introduced here.Follow-ups raised during review
oneOfbranch, so a standalone agents registry (spec 014 FR-004) cannot validateNODE_OPTIONSNote: Qodo's subscription lapsed during this work, so parts of the automated review coverage were unavailable.
refactor: changelog-agent - establish quality framework and spec 015 …