Skip to content

Put the hallucination-hunter agent where it can actually be found - #3

Merged
AlveeeRahman merged 1 commit into
mainfrom
docs/hallucination-hunter
Aug 19, 2026
Merged

AlveeeRahman merged 1 commit into
mainfrom
docs/hallucination-hunter

Conversation

@AlveeeRahman

Copy link
Copy Markdown
Owner

The hallucination-hunter agent has shipped since 1.1.1 and was almost invisible — two prose sentences, and nothing else. Not in either flowchart, not on the Pages site, not in the repository description. A subagent nobody can see is a subagent nobody runs.

Why the map never showed it

skill_mapper.py builds its graph from resolved markdown links in SKILL.md. SKILL.md named the path in backticks — which is text, not a link. So the only route to the file was through README.md, and README.md is not on the path Claude follows under progressive disclosure.

The file was reachable by a human browsing GitHub, and not by the agent that was supposed to use it.

Making it a real link takes the map from 35 resolved links to 36 and puts SKILL.md one hop from the agent. Verified with the tool, before and after:

before:  31 files mapped · 35 resolved links   →  n_README_md --> n_agents_hallucination_hunter_md
after:   31 files mapped · 36 resolved links   →  n_SKILL_md   --> n_agents_hallucination_hunter_md
                                                  n_README_md  --> n_agents_hallucination_hunter_md

That is the same class of defect skill_mapper.py exists to surface, found in this repository by running this repository's own tool on itself.

The conceptual flowchart was also wrong

It said "six independent inspections" and drew five — claim_auditor.py was missing from the diagram entirely. Both it and the agent are now in it, with the agent placed where it actually sits: downstream of claim_auditor, taking its UNVERIFIED list as a worklist rather than starting from prose.

The "genuine output" block had drifted

Regenerated from the current tool rather than hand-patched — the node-id scheme changed at some point (SKILL_md → n_SKILL_md) and the committed block still used the old one. It stays trimmed to README size, and says so.

New: a features table

The README had no at-a-glance summary. Added one covering all seven scripts and the agent, with a column saying what each one answers rather than what it is called — and marking plainly which one is not a script.

Pages site

Gains a "Where the scripts stop" section explaining the boundary the agent exists to cross: a deterministic check can prove a documented flag does not exist, but not that a cited paper fails to say what it is cited for, that a version went stale, or that a benchmark number has no source. Those need judgement, and judgement is where a model hallucinates — so the agent is built to make guessing expensive: tools first, evidence per finding, UNVERIFIED rather than a guess, and a closing statement of what it did not check.

Also adds the agent to the resources list and to .github/repository-metadata.yml.

Checks

  • claim_auditor.py → TRUTHFUL (0 contradictions)
  • script_tester.py → exit 0
  • pytest tests → 196 passed
  • JSON-LD parses; meta description 163 chars

scripts/ and tests/ are untouched — this is documentation and one link.

🤖 Generated with Claude Code

The agent has existed since 1.1.1 and was almost invisible. It appeared in two
prose sentences and nowhere else: not in either flowchart, not on the Pages
site, not in the repository description. A subagent nobody can see is a subagent
nobody runs.

The interesting part is why the map did not show it. skill_mapper.py builds its
graph from resolved markdown links in SKILL.md, and SKILL.md named the path in
backticks -- which is text, not a link. So the only route to the file was
through README.md, and README.md is not on the path Claude follows under
progressive disclosure. The file was reachable by a human browsing GitHub and
not by the agent that was supposed to use it. Making it a real link takes the
map from 35 resolved links to 36 and puts SKILL.md one hop from the agent.

That is the same class of defect skill_mapper exists to surface, found in this
repository by running this repository's own tool on itself.

The conceptual flowchart in the README was also wrong in a smaller way: it
claimed six independent inspections and drew five. claim_auditor.py was missing
from it entirely. Both it and the agent are now in the diagram, the agent shown
where it actually sits -- downstream of claim_auditor, taking its UNVERIFIED
list as a worklist rather than starting from prose.

The "genuine output" mermaid block was regenerated from the current tool rather
than hand-patched. It had drifted: the node-id scheme changed at some point and
the committed block still used the old one.

Adds a features table, which the README did not have, covering all seven scripts
and the agent, and says plainly which of them is not a script.

The Pages site gains a "Where the scripts stop" section explaining the boundary
the agent exists to cross: a deterministic check can prove a flag does not
exist, but not that a citation fails to support the claim it is attached to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@AlveeeRahman
AlveeeRahman merged commit 364e6a1 into main Aug 19, 2026
5 checks passed
@AlveeeRahman
AlveeeRahman deleted the docs/hallucination-hunter branch August 19, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant