Skip to content

Design Doc Diff Checker: tighten tool fence, setup gate and listing claims #46

Description

@anildukkipatty

Follow-up from the post-merge review of #38. The bot is live in the library — nothing here is a security concern or a suggestion that anything was done in bad faith, but several items will affect people who install it. @arjunKumbakkara would you be up for a follow-up PR?

Ordered roughly by how many installers each one affects.

1. allowedTools uses Claude Code tool names, but agent is opencode

bot.json sets:

"agent": "opencode",
"allowedTools": ["Read", "Grep", "Glob", "Bash"]

opencode's tools are lowercase (read, grep, glob, bash). The two existing bots using that exact capitalised list — pr-guardian and shipguard — are both claude-code. Depending on how the fence is matched, this either matches nothing (the bot can use no tools at all) or is ignored entirely (no fence). Either way it isn't doing what it looks like it's doing. Worth confirming against gitbot's fence implementation and lowercasing if that's what opencode expects.

2. setup.md gates the whole bot on a global pip install

Per docs/bot-schema.md, a bot with a setup.md refuses all work until setup reports SETUP_COMPLETE. So someone diffing two Markdown files with no PDF anywhere is still blocked until python3 -m pip install pypdf succeeds — and an unpinned global install with no venv fails outright under PEP 668 (Debian/Ubuntu system Python, Homebrew Python). The PR description says pypdf is needed "only if the user wants PDF support", which is the better behaviour; setup.md currently makes it unconditional.

Suggestion: drop setup.md entirely and make step 3 of the instructions degrade gracefully — if pypdf isn't importable, say so and carry on with the text formats.

3. Unrestricted Bash behind a read-only promise

about says "all without touching your files" and there's no disallowedTools. Unrestricted Bash is a write and arbitrary-execution path (python3 -c, > redirection), so the read-only guarantee rests on a prose rule in instructions.md rather than on a control.

shipguard is a useful reference here: same plan mode, same allowlist, but it adds a disallowedTools list blocking Bash(python3 *), Bash(pip *), Bash(*>*) and reads of .env/keys/credentials. This bot genuinely needs to run Python for PDF extraction, so the fence wants to be narrow rather than absent — ideally just the one pypdf extraction path.

4. Document-sourced instructions

The bot's entire input is documents it selects itself, including PDFs, while holding Bash. Nothing currently tells it to treat document content as data rather than as instructions. One line in the Rules section covers it.

5. File selection matches more than design docs

Step 1 matches any .md / .yaml / .yml / .json / .txt / .pdf in the folder, excluding only READMEs, CHANGELOGs and lockfiles. In a real project folder that also matches package.json, tsconfig.json, CI yaml, values.yaml, secrets.yaml. It then picks the two most recently modified and reads them — and it announces the selection after reading, with reads unprompted under plan mode.

Suggestion: require the filename to contain design, spec or openapi, or confirm the two picks before reading them.

6. "Validate OpenAPI 3 spec conformance" promises more than the bot has

There's no validator and no access to the TM Forum guidelines or the OpenAPI meta-schema — it's model recall. The instructions already handle this well ("do not invent content", "if a standard check cannot be performed, say so"), but the feature bullet still reads as mechanical validation. Softening it to "review against" / "flag likely gaps" would match what it actually does. Confident-but-wrong compliance verdicts are the failure mode to design against, especially for TM Forum.

7. Unverified model slug

Worth double-checking opencode/nemotron-3.5-lightning-free resolves. If the slug is wrong, the bot fails at install for everyone.


For the record, everything mechanical was in spec at merge: field lengths, exactly three features, valid mascot enums, slug matching the folder name, author matching the PR author, one bot per PR, no absolute paths or credentials in the diff.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions