Skip to content

Build the CI validator and index.json generator the docs already specify #48

Description

@anildukkipatty

docs/bot-schema.md and CONTRIBUTING.md both describe a CI validator as though it exists — "a required field turns that silent runtime blank into a loud CI failure on the pull request, before you ever look at it", "Validation catches the mechanical things. Review is for the two questions a schema cannot answer." None of it is built yet. cc @anildukkipatty

Missing today:

  • .github/workflows/ — does not exist, so no checks run on any PR
  • schema/bot.schema.json — does not exist, yet all 12 bots point $schema at ../../schema/bot.schema.json, so every bot.json has a dangling reference and no editor completion
  • index.json — described as "generated by CI ... the only file the app fetches". Not generated, so nothing consumes the library yet
  • verified.json + CODEOWNERS — the docs put the verified badge in a maintainer-only root file. Neither exists, so the badge has no source

The practical effect is that all twelve mechanical checks fall on whoever reviews the PR, by hand, every time. #46 and #47 are both partly a consequence of that.

Rules the docs already commit to

Fail the PR when:

  1. a required field is missing or empty
  2. slug != folder name, or collides with another bot
  3. name collides with another bot
  4. features is not exactly 3, or any length limit is exceeded (name ≤24, description ≤80, about 120–400, features ≤60 each, examplePrompt ≤200, slug matching ^[a-z0-9]+(-[a-z0-9]+)*$)
  5. category, mascot.body, mascot.color, mascot.activity, agent or permissionMode is outside its enum
  6. agent is opencode and model is absent
  7. instructions.md is missing, empty, or over ~600 words
  8. author.github does not match the PR author
  9. the PR touches more than one bot folder, or edits index.json / verified.json
  10. the diff contains an absolute home-directory path or anything shaped like a credential

Warn (not fail) on a missing mascot block — the docs are explicit that this is a nudge in review, not a break.

On merge to main, regenerate index.json: every bot.json field, plus instructions and setupInstructions inlined from the Markdown, plus derived authorPhoto, the resolved colour variable, and verified from verified.json.

Two bots on main would fail rule 7 today

I dry-ran the list above against current main:

Bot Result
shipguard FAIL — instructions.md is 1392 words vs the ~600 limit
npm-publisher FAIL — 650 words
lumie, merge-safety-reviewer, npm-publisher auto-approve with no allowedTools fence
everything else passes

So the validator needs a decision before it can go green: either grandfather what's already merged, or fix those two. Worth noting shipguard's length is mostly its very thorough disallowedTools list, which is a good thing — the word limit applies to instructions.md only, so it's the prose that needs trimming, and it may be that 600 is simply the wrong number.

The three auto-approve-without-a-fence bots aren't a documented failure, but CONTRIBUTING.md says "pair it with a tool allowlist ... or send it back", so surfacing them as a PR warning would put the right pressure on new submissions.

Three checks worth adding, from #46 and #47

These aren't in the docs but each would have caught something a human reviewer nearly missed:

  • Tool names must match the agent's namespace. Add Design Doc Diff Checker #38 shipped "agent": "opencode" with "allowedTools": ["Read","Grep","Glob","Bash"] — Claude Code's capitalisation. opencode uses lowercase tool names, so that fence likely matches nothing or is ignored outright. A cross-check of tool names against the declared agent is cheap and catches a class of silently-broken fences.
  • model shape, and ideally resolvability. provider/model shape is trivial to assert. A wrong slug means the bot fails at install for every user, so even a shape check beats nothing.
  • Flag setup.md for reviewer attention. Per the docs a setup.md makes the bot refuse all work until setup succeeds, so an unnecessary step disables the bot rather than degrading it. 7 of 12 bots ship one. A warning prompting "is this genuinely required?" is in the spirit of the existing guidance.

Soft/style warnings, low priority: description should be one sentence (standout's is three, within the char limit).

Sequencing

There are 6 open PRs (#40–#45), all one contributor's "Ghost of <maintainer>" series with near-identical shape. Landing the validator before that batch means it gets exercised on six similar PRs at once, and they get checked consistently rather than by reviewer stamina. That's the argument for doing this next rather than after.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or requesthelp wantedExtra attention is needed

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions