Physics-verification layer for CatBot outputs (catgo_verify) — clean resubmission of #541 - #546
Open
leshenzhang wants to merge 20 commits into
Open
Physics-verification layer for CatBot outputs (catgo_verify) — clean resubmission of #541#546leshenzhang wants to merge 20 commits into
leshenzhang wants to merge 20 commits into
Conversation
An additive verification layer that catches SILENT errors in agent-produced
computational-chemistry results — runs that exit 0 with normal-looking
numbers and wrong physics (PAW/POTCAR reference mismatch, ZPE zero-fill,
corrupted Hessians invisible to n_imag, stale convergence flags, scheduler
COMPLETED on crash, residual imaginary modes, out-of-range limiting
potentials, MLIP force errors masked by absolute RMSE, ...).
Two new stdlib-only modules + hooks; no existing tool logic changed:
- verify_gates.py — 18 physics gates over 16 silent-error classes, each
threshold traced to a recorded real failure; a verifiability layer that
refuses to certify a claim whose provenance is absent or EMPTY
(UNVERIFIABLE, default-deny for unregistered claim types); and
workflow_consistency for cross-step DAG checks (POTCAR set identical,
ENCUT non-decreasing, k-grid consistent). Absent inputs report SKIP,
never silently dropped.
- verify_enforcement.py — tri-state {allow/prompt/forbidden} decision
(design borrowed from openai/codex execpolicy, Apache-2.0), strictest-
wins. Numeric-producing tools are matched by PREFIX so BOTH server
variants are guarded (merged catgo_analyze/catgo_catalysis AND the
61-tool fine-grained catgo_dos_*/catgo_catalysis_*/catgo_cohp_*/
catgo_bands_*/catgo_energy/catgo_freq_*). A produced number marks the
session unverified; only a catgo_verify run in which at least one gate
actually ran clears it (an empty all-SKIP verify cannot bypass the gate);
an HPC submit while unverified returns BLOCKED with instructions.
- catgo_verify tool wired into both server_claude_code.py (merged variant)
and server.py (61-tool variant that catbot-plugin's mcp_server.py
delegates to), so the plugin route is covered.
- catbot.md playbook: verify-before-trust rule + usage section.
- test_verify.py (14 tests incl. regressions for prefix matching, the
empty-verify bypass, and empty-provenance certification) + a scoped
verify-python.yml CI workflow.
Evaluation on an information-isolated corpus of 79 real accepted results
and 40 real recorded failures: frozen-gate catch-rate 48% (Wilson95
[33%,63%]), corpus-informed 80% [65%,90%], false-alarm 8% [4%,16%].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by driving the live MCP server as an agent, not by the unit tests:
catgo_catalysis -> numbers produced
catgo_workflow run -> BLOCKED (unverified) correct
catgo_verify -> "FAIL ... do NOT report this result as correct"
catgo_workflow run -> allowed WRONG
Clearing the pending state keyed on "a gate ran", so the gate forced the
agent to look, not to pass: a result the layer had just declared wrong
could still spend HPC budget. Same shape as the bug class this layer
exists to catch — it exited 0 and looked fine.
- mark_verified() now clears only when a gate ran AND nothing failed and
no declared claim was left uncertified; a later clean audit on the
fixed result clears it.
- A FAIL is waivable only through register_override(): the exact failing
gate names, a >=20-char justification, one-shot, recorded in an audit
list, invalidated by any new failing audit. False alarms cost a logged
waiver instead of a dead end; the PROMPT tier stamps the response.
- Escape hatch for the no-coverage case (no gate could check the result):
same narrow waiver under the sentinel gate 'no-coverage'. Without it
such a session is blocked forever — nothing failed, nothing to fix.
- verify_enforcement.py's own __main__ self-test was stale and exited 1
(it asserted postmark('catgo_verify') clears, which the R2 fix moved to
mark_verified). Fixed and extended to the new state machine.
Both MCP variants: schema (override/justification) + PROMPT stamping in
server.py and server_claude_code.py.
Tests: pytest test_verify.py 19 passed (14 -> 19, +R5 regressions);
standalone runner and both module self-tests green; a scripted probe over
real MCP stdio passes 15/15 across bad-result, clean and no-coverage
paths. Gate behaviour is untouched — the independent-corpus ladder is
unchanged at 0/10/52/80 with 8% false alarms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
Measured on the information-isolated corpus, accepted results reach U_L = +6.550 V (strong-binder regime, geometry verified in the source project) while a reference-mismatch artifact sits at +6.346 V. The two populations overlap, so the zero-false-alarm ceiling of ANY window is 2/4 catches; the deployed [-2.0, 3.5] window bought 2 catches and was the sole source of every false alarm the suite had (6/79). Moving the threshold would be fitting the test set. What separates the populations is not magnitude but which reaction, which reference state and which PCET convention produced the number — provenance. So U_L becomes the claim `limiting_potential` in the verifiability layer: undeclared -> UNVERIFIABLE, declared -> checkable. Suite is now 17 gates over 15 taxa. On the independent corpus the baseline ladder becomes 0% / 10% / 52% / 75%, with 0% false alarms at every rung (was 80% catch at 8% false alarms). Detection drops by 2 cases; prevention does not — both cases are still blocked from reaching an irreversible tool, now as uncertified claims rather than gate FAILs. pytest 19 passed; module self-tests green; live MCP probe 17/17. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
…nt, restraint sign)
Mined from the same failure-mode survey. All identities, no thresholds:
dos_electron_count integral(DOS) up to E_F == NELECT — catches dropped
k-point weights (Cr magmom 74 uB vs a true 2-3) and a
truncated DOS window (73 uB entirely from s spin-up)
relaxation_minimum_endpoint E_final == the minimum the run itself visited — catches
a force criterion firing on a shoulder (endpoint 24 meV
above a geometry already passed through)
restoring_force_sign a restraint must reduce |r-r0| — catches a sign error
that pushed an ion from r0=3.0 A out to 6.13 A
26 gates over 24 taxa. The nine gates added today are marked PROVISIONAL_GATES:
proven on their recorded case in the self-test, but no held-out corpus carries
their input fields yet, so they are reported separately from the dead-gate rate
rather than counted as capability.
Generated from agentbench/verifier.py via sync_copies.py; pytest 19 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
… dead
Three modules computed the server root with one dirname too few:
mcp_tools/server_claude_code.py, mcp_tools/server.py, mcp_tools/tools/structure.py
all inserted server/catgo/ at sys.path[0] instead of server/. Since
server/catgo/workflow/ exists, it shadowed server/workflow/, so every
`from workflow.catalysis... import` raised "No module named 'workflow.catalysis'"
and catgo_catalysis answered "Catalysis module not available: ..." for every call
in the merged MCP variant — a dead tool wearing a plausible error message. Found by
driving the live server, not by reading the code.
Also adds the provenance envelope for numeric returns (provenance.py, applied at the
catgo_catalysis return). Motivation, measured rather than asserted: of 94 real
failures from 17 projects, 28% were classes the gates already cover and were missed
only because the record lacked the field; ranked by failures unlocked the top fields
are `energy` (18) and `n_atoms` (12). An independent audit of 178 numeric-returning
tools across 18 public comp-chem agent repos found 59.6% emit no core method
provenance and 0/178 declare a sign convention.
The envelope carries {value, provenance{tool, action, inputs, method, unit,
convention}} and — the load-bearing part — `unverifiable_without`: the fields the
tool does NOT have and therefore cannot vouch for, which the verifiability layer
turns into UNVERIFIABLE instead of a silent pass. It never invents a field; a
None-valued argument counts as absent.
Tests: pytest 24 passed (19 + 5 new), three standalone self-tests, live MCP probe
17/17. CI runs the new files too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
Found by a new live probe scenario that tests whether the provenance envelope actually closes a loop rather than just returning more text. mark_verified() returned early on `not covered`, discarding uncertified_claims with it. So for a result no value gate can check — exactly the case the verifiability layer exists for — a refused claim left no trace: the submit was still blocked, but by the vaguer "N results have not passed catgo_verify" instead of naming the claim, and the refusal never reached the session state. Now a refusal is recorded whether or not a gate ran; a genuinely empty verify (nothing ran, nothing refused) still does not clear, as before. With this, the loop closes end to end on the live server: catgo_catalysis declares unverifiable_without=[ul_reaction, ul_reference, ul_convention] -> catgo_verify refuses to certify limiting_potential -> the submit is blocked naming claim:limiting_potential -> the agent supplies those three fields -> the same value certifies and the submit passes. pytest 25 passed; live probe 22/22 across four scenarios. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
…e criterion Two changes, both found by running against real data rather than fixtures. 1. force_convergence was judging every run against EDIFFG=-0.02 when the INCAR did not set one. On 120 real VASP dirs on Shaheen it fired 117/117 — a 100% fire rate is a broken gate, not a discovery. The sample was an MLIP training set of displaced and defective geometries, where large forces are the point. EDIFFG is now required (absent -> declared SKIP, never a made-up default), and the gate declines runs that do not relax at all (IBRION<0, NSW<1) or that set an energy criterion (EDIFFG>0). New _na() helper: inapplicable is SKIP, never PASS. 2. precheck_inputs() adds six pre-submission gates over INCAR/KPOINTS/POTCAR — one-tag-per-line, KSPACING-vs-KPOINTS, NSW-for-optimizer, LMAXMIX-covers-valence, non-magnetic MAGMOM, required keys. Unlike the output-side gates these are not capped by what a pipeline emits: the agent writes the INCAR, so the fields are there. Measured on 5624 real INCARs across 20 projects they decided on 242-5624 files each, against 19% coverage for the output-side gates. The one-tag gate initially fired on 569 files (10%), but multi-tag lines only break the vasp.6.4.2-cp build; stock VASP parses them fine. It is now conditioned on the binary, and an undeclared binary is a SKIP naming what is missing — which turns 569 files into a measurement of an input-side provenance gap rather than 569 false alarms. Also fixes MAGMOM 'N*value' parsing, which read the repeat count as a moment. verifier self-test 26 gates + 6 input gates; pytest 25 passed; 4 copies in sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
… it referenced Measured on a live ai-screen tree on Ibex: of 751 adsorbate directories across five trees, only 67 (9%) could be paired with their bare slab by name heuristics. One project used two opposite conventions in its own trees — <species>_<M1>_<M2> in aiscont, <M1>_<M2>_<species> in aisn2nnh — with inconsistent metal-pair ordering on top. A four-way matcher (drop leading / drop trailing, both orderings) rescued one tree from 0 to 13 pairs and still stops at 9%. That matters because the pairwise PAW gate carries the largest recorded blast radius here: a Hf_pv/Hf_sv split between the ads and bare directories shifted 122 units by about 3.1 eV and survived two months precisely because every single-directory check passes. The gate is fine; it simply cannot run without knowing what was referenced. So NEEDS['binding_dG'] now asks for reference_dir. The emission side deliberately asks for more than certification requires, and the reason is in the comment. verifier's PROVENANCE_SPEC is untouched so no already-measured number moves. On the 67 pairs that could be checked, paw_consistency and the workflow gates fired zero times — consistent with that failure having been fixed. 684 directories remain unchecked, so this is not a clean bill of health for the tree. provenance self-test, test_server_paths 5 tests, pytest 25 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
…ourselves on it The audit we published found 59.6% of 178 public comp-chem tools emit no method provenance, with discard at the payload boundary as the dominant cause. Publishing that with one instrumented call site of our own was the most obvious hypocrisy attack available, so the envelope now lives at the dispatcher in both MCP variants: any JSON payload from a numeric tool is wrapped once, prose and error strings are left alone, and an already-enveloped payload is not double-wrapped. Claim types registered for the numeric families, each justified by a recorded failure: charge (Bader partitioning depends on the PAW set and electron count), electronic_structure (a DOS-derived quantity needs applied k-point weights and a window covering both spin channels — an unweighted sum and a truncated window each produced a ~73 uB moment here), frequencies (n_modes == 3*n_free catches a POSCAR missing Selective dynamics). eval_self_emission.py scores CatGo on the same 10-field checklist, and the honest answer is unflattering: across 16 numeric tools / 21 payload shapes we emit a source pointer on 21/21 and ZERO actual physics fields. What changed is that 16/21 payloads now NAME what they cannot vouch for, where the external audit found 0/178 with any such concept. Solvation and units are declared by nothing at all — a real gap, left visible rather than smoothed over. pytest 25 passed; live probe 22/22; the four gate copies remain in sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J6v95zAFDiSkrWvNZfTZXA
Review findings addressed (on top of the digest-identity rework): - catgo_workflow_engine read/control actions (status, list, get_result, get_dag, pause, create, add_task, modify_params, reset) are no longer treated as submits; only submit/run/execute/start/retry/resume are gated - numeric families extended: catgo_kmc*, catgo_md_*, catgo_optimize, catgo_cn_coupling*; geometry-only actions (adsorption_sites, dft_input, symmetry) no longer arm a pending state nothing can verify - gas_entropy_included=False now FAILs the audit (gate gas_entropy_declared, taxon G) instead of scoring PASS + VERIFIABLE on field presence alone - _response_succeeded also rejects "workflow_engine error:" and "Unknown <tool> action" formats so failed runs cannot arm as unverified - CI executes the module self-tests (python verify_enforcement.py / provenance.py) that the review found were never run Also lands the verification-layer rework this branch accumulated locally: per-result sha256 digests + identity supersedence (verify one no longer releases all; unbound verifies are ignored and audited), provenance envelopes for every numeric payload, PROVENANCE-ONLY verdict, narrow one-shot no-coverage/FAIL overrides, pre-submit VASP manifest -> collector lineage, and the focused CI regression gate (152 tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sfv8Lxuyyzvh8PP3X1t8CR
…entories Merging current main (base was 125 commits behind) surfaced two registry tests added on main that the stale base never saw: every workflow must carry a release-surface classification (license-policy) and a publisher classification (version-consistency). verify-python.yml is validation-only: deploymentOrValidationOnly + non-publisher (no PUBLISHER_SIGNAL match). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sfv8Lxuyyzvh8PP3X1t8CR
…claim
Driving the real MCP server (live_mcp_probe, now 29 checks / 5 scenarios)
found two ways a NORMAL call still reached a verify-forever/blocked-forever
state after the digest rework:
- a bare-dict verify that FAILed set legacy_failed, and no later clean
re-verify could clear it — the FORBIDDEN message's "a clean audit clears
this" was false; the fix-and-reverify loop dead-ended in override-only.
mark_verified now clears the identity-less failure state on any clean
covered audit (produced-result pending stays digest-strict).
- every real oer/co2rr/nrr call declared ul_reaction/ul_reference/
ul_convention unverifiable, and because the digest binds {value, claim,
provenance} the agent could not supply them without breaking the binding:
certification was override-only. The producer now emits them (CHE module
properties, potentials vs RHE) — the loop closes at emission.
- claim types provenance.py emits (free_energy, d_band_center, charge,
electronic_structure, frequencies) were absent from PROVENANCE_SPEC, so a
fully provenanced result still scored UNKNOWN-CLAIM (default-deny) —
override-only for those tool families. Registered, mirroring
provenance.NEEDS, with a guard test (R12) enforcing the pairing.
Tests: R10 (bare FAIL -> clean re-verify clears), R11 (real oer envelope
certifies and clears end-to-end, zero overrides), R12 (registry pairing).
155 pytest + module self-tests green; live probe 29/29 x3 runs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sfv8Lxuyyzvh8PP3X1t8CR
leshenzhang
added a commit
that referenced
this pull request
Jul 30, 2026
Merge-readiness audit finding. The repo has no workflow that runs pytest at all (grep -rl pytest .github/workflows -> nothing on main), so the seven suites this branch adds — and the Python tests that were already in server/ — would land as a local-only safety net that CI never exercises. Runs the agent->viewer publication suites plus the two pre-existing view_state routing tests: 89 tests, and viewer_publish's own module self-test. Registered in both workflow inventories (license-policy: release-surface, version-consistency: publisher classification). PR #546 was blocked by exactly these two gates for adding a workflow without registering it. If #546's verify-python.yml lands first, consolidate the two — they overlap in intent, not in file list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4 EOF
leshenzhang
added a commit
that referenced
this pull request
Jul 30, 2026
Merge-readiness audit finding. The repo has no workflow that runs pytest at all (grep -rl pytest .github/workflows -> nothing on main), so the seven suites this branch adds — and the Python tests that were already in server/ — would land as a local-only safety net that CI never exercises. Runs the agent->viewer publication suites plus the two pre-existing view_state routing tests: 89 tests, and viewer_publish's own module self-test. Registered in both workflow inventories (license-policy: release-surface, version-consistency: publisher classification). PR #546 was blocked by exactly these two gates for adding a workflow without registering it. If #546's verify-python.yml lands first, consolidate the two — they overlap in intent, not in file list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4 EOF
leshenzhang
added a commit
that referenced
this pull request
Jul 30, 2026
Merge-readiness audit finding. The repo has no workflow that runs pytest at all (grep -rl pytest .github/workflows -> nothing on main), so the seven suites this branch adds — and the Python tests that were already in server/ — would land as a local-only safety net that CI never exercises. Runs the agent->viewer publication suites plus the two pre-existing view_state routing tests: 89 tests, and viewer_publish's own module self-test. Registered in both workflow inventories (license-policy: release-surface, version-consistency: publisher classification). PR #546 was blocked by exactly these two gates for adding a workflow without registering it. If #546's verify-python.yml lands first, consolidate the two — they overlap in intent, not in file list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4
…on the
path production actually uses
P0 — the HTTP transport ran no verification at all. `routers/mcp_http.py` is
what the desktop app and every HTTP client talk to. It listed `catgo_verify` in
TOOLS, answered "Unknown tool" for it, and called neither precheck nor postmark,
so numeric results never armed and HPC submits were never gated there. The
wrapper existed only in the stdio server. It is now extracted as
`run_with_verification` and used by both — two copies would drift again.
`catgo_validate_config` had the same missing branch; the review named one tool,
the hole covered two.
The gate was blind to the failure it was built for. `gate_incar_tag_echo_identity`
skipped any requested tag absent from the OUTCAR echo — but the cp build drops
every tag after the first on a line, so a dropped tag never reaches the echo.
The gate reported "all N requested tags echoed" for exactly the case in its own
docstring. Absent is now a mismatch.
One energy window cannot judge two families of code. `E_SANE_PER_ATOM` is a
valence-only (PAW/pseudo) window; an all-electron code lands hundreds of eV/atom
lower, so a legitimate ORCA -2080.5 eV on 3 atoms was FAILed and then locked HPC
submission behind a "known bad result". The gate now takes `code`, and without it
SKIPs — it must not judge against physics it was not told.
`catgo_workflow_engine.get_result` returns the node's numbers and was not armed,
so an agent could read a result and submit the next job unverified. `status`
stays exempt: it carries progress, not results.
`shlex.quote("~/calculations")` makes the tilde literal, so the default work root
resolved to a directory that does not exist and no TOTEN/manifest metadata came
back. Quote after the tilde instead.
CatGo's own Shaheen3 template ends in `echo "Calculation finished on $(date)."`,
which the new validator rejected twice over — blanket `$(` ban plus an
allowed-setup list without `echo`. Both scoped to non-execution lines; command
substitution on an execution line is still rejected, with a test.
One process now serves every HTTP client, so verification state is keyed by
panel id instead of a global "default" — otherwise one user's unverified result
blocks another's submit and an override can be spent by a different session.
PROMPT does not wait for a human and never did; documented at the definition
rather than implied by the name. Real approval lives in the permission layer
above this one.
`pyproject.toml` test extra asked for `httpx2`, which nothing imports.
Tests: +19 in test_review_findings.py, each named after the production failure
it prevents. Branch regression 64 failed -> 51 failed / 1384 passed: 13 of the
branch's own pre-existing failures are fixed and none introduced.
Not addressed here, and why: custodian-vs-manifest hashing is a design conflict
(self-healing rewrites INCAR/KPOINTS while the collector demands identical
hashes) that needs a decision on what "the same run" means, not a patch;
precheck_inputs() is still only called from tests; KSPACING-only jobs still fail
the KPOINTS preflight.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4
…skipping every real VASP result The verify-layer-python job failed on `AttributeError: 'Server' object has no attribute 'list_tools'`. Same cause as on the visualisation branch: this workflow hand-wrote its pip list, `mcp>=1.0.0` has no upper bound, pip resolved 2.0.0, and that release drops `Server.list_tools` — used as an import-time decorator in catgo/mcp_tools/server.py, so the module cannot be imported at all. The cap now lives in requirements.txt and pyproject.toml, where a fresh install is protected too, and the job installs what the server declares. Confirmed the cap reaches the built wheel's metadata (`mcp<2,>=1.0.0`). Building a throwaway venv the way CI does then caught a regression my dev environment could not: requiring a literal `code` on the energy gate made every real VASP result SKIP, because a VASP payload carries POTCAR titles rather than a code name, and two round-trip tests dropped from PASS to PROVENANCE-ONLY. The family is now inferred — explicit `code` wins, `potcar_titels`/`ads_titels` mean a PAW/pseudopotential run — and the gate returns SKIP itself when the result says neither, instead of judging against physics it was not told. All four cases are asserted: VASP passes, VASP garbage fails, ORCA passes, a bare energy skips. Verified in the clean venv exactly as CI runs it — module self-tests, the focused regression gate (174 passed), and the wheel build + import smoke. Branch regression 64 failed (pre-existing baseline) -> 49 failed / 1386 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4
NOT ready for the PR branch. Pushed only so the work survives the scratch worktree; verified state is recorded below. Implemented (code believed correct, see caveats): * KSPACING — the submit-side preflight no longer demands KPOINTS when the INCAR sets KSPACING, and `catgo_emit_input` marks that entry `mandatory:false` so the collector agrees. Previously a valid KSPACING job was rejected at submit. * custodian vs manifest hashes — the manifest now records `use_custodian`, and the collector treats a divergence in INCAR/KPOINTS/POSCAR under custodian as recorded provenance (`custodian_rewritten_inputs`: submitted vs ran) rather than tampering. POTCAR stays strict always; without custodian nothing may change. Semantics chosen: the manifest records what was SUBMITTED, and a self-heal is a legitimate, recorded rewrite. * precheck_inputs was dead code — `write_vasp_input_manifest` now reads INCAR and KPOINTS back and runs it (`_audit_vasp_inputs`). Advisory: it logs, it does not block, because the output-side layer owns refusal. * postmark race — a numeric call is armed BEFORE dispatch (`arm_pending`), so a concurrent submit during a multi-second HPC call can no longer slip through. postmark retires the marker whether the call succeeded or failed, so a failed tool cannot wedge the session. Module self-test passes. Caveats — why this is not on the PR branch: * tests/test_review_findings.py: 27 pass, 2 fail. The failures are my test fixture, not the code: I first wrote the manifest checks by RE-IMPLEMENTING the collector's comparison, which only proved the test agreed with itself. Rewriting them to drive the real `_parse_vasp_metadata` immediately exposed that the fake manifest omits fields the parser requires — it reports `binary_declared` and `missing_mandatory_inputs`. The custodian rewrite path itself already works there (`custodian_rewritten_inputs` is populated). * The full branch regression and the clean-venv CI reproduction have NOT been run since these edits. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GsCFrY7yAHpUzCa4825HW4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Clean, squashed resubmission of #541 with all four review findings fixed and test-verified. Single commit on top of current
main.What
An additive verification layer that catches silent errors in agent-produced computational-chemistry results — runs that exit 0 with normal-looking numbers and wrong physics (PAW/POTCAR reference mismatch shifting ΔG by ~3 eV while geometry passes QC; corrupted Hessians invisible to
n_imag; ZPE zero-fill; stale convergence flags; schedulerCOMPLETEDon crash; residual imaginary modes; out-of-range limiting potentials; MLIP force error masked by absolute RMSE).Contents (7 files, +978 lines, no existing tool logic changed)
verify_gates.py— 18 physics gates over 16 silent-error classes (every threshold traced to a recorded real failure) + a verifiability layer: a claimed quantity whose provenance is absent or empty is flaggedUNVERIFIABLE(default-deny for unregistered claim types) +workflow_consistencycross-step DAG checks. Absent inputs reportSKIP, never silently dropped.verify_enforcement.py— tri-state{allow, prompt, forbidden}, strictest-wins (design borrowed from openai/codex execpolicy, Apache-2.0). Numeric tools matched by prefix, covering both server variants (mergedcatgo_analyze/catgo_catalysisand the 61-tool fine-grainedcatgo_dos_*/catgo_catalysis_*/catgo_cohp_*/catgo_bands_*/catgo_energy/catgo_freq_*). A produced number marks the session unverified; only acatgo_verifyrun in which at least one gate actually ran clears it (an empty all-SKIPverify cannot bypass the gate); an HPC submit while unverified returnsBLOCKED.catgo_verifytool wired into bothserver_claude_code.pyandserver.py(the variantcatbot-plugin'smcp_server.pydelegates to), so the plugin route is covered.catbot.md— verify-before-trust playbook rule + usage section.test_verify.py— 14 tests incl. regressions for the four Add a physics-verification layer for CatBot outputs (catgo_verify) #541 review findings;verify-python.yml— scoped CI (standalone stdlib run + pytest), already green on this branch.Review findings from #541 — all addressed here
catgo_catalysis_oerarms the gate, unverified submit isFORBIDDEN).catgo_verifyclears the unverified state → clearing moved tomark_verified(), called only whencoverage.ran > 0._group_presentrequires a non-empty value.test_verify.py+verify-python.yml(green).Evaluation
On an information-isolated corpus of 79 real accepted results + 40 real recorded failures: frozen-gate catch-rate 48% (Wilson95 [33%, 63%]), corpus-informed 80% [65%, 90%], false-alarm 8% [4%, 16%].
Supersedes #541.
🤖 Generated with Claude Code