docs: make the repo agent-friendly (Claude + Codex), and fix what was stale

[fcostaoliveira]

Split out of #319, where it had no business being — this touches no dataset, engine or benchmark code.

The instructions were wrong in ways that mislead an agent

AGENTS.md opened with "Supports Redis (RediSearch) and VectorSets engines" and "only Redis and VectorSets are ported". The repo has 15 engines, and the table four lines below already listed eleven of them. An agent reads the overview first and builds its model of the codebase from it, so this is a correctness fix rather than tidying.

Two of the Workflow Rules were self-defeating:

• Rule 1 said "always run make check" — but make check is fmt-check + lint and runs no tests. Following the instructions exactly produces a green local run that CI can fail.
• Rule 5 said "never bypass make targets" — but dragonfly, kividb, chroma and vertex had integration tests with no make target, so the rules made those four suites unrunnable.

make agent-check

One command that reproduces the entire no-Docker CI gate, in CI's order and profile:

cargo fmt --check
cargo clippy --all-targets -- -D warnings
cargo test --lib --bins --release
cargo test --test {integration_cli,overhead_invariants,harness_invariants} --release

If it passes, the non-Docker CI jobs pass. make check and make help now both say what check does not cover.

--release is called out deliberately. CI runs the suites in release, so a debug_assert! does not exist there — which is how a guard ends up absent from precisely the binary that ships. Cheap to state, expensive to rediscover.

Also

• Added the four missing integration-test-* targets, matching CI's ports and env vars. Vertex is listed explicitly as cloud-only, so its absence from the Docker targets reads as a documented fact rather than an oversight.
• Refreshed the structure block — it predated the sparse/multivector readers, query_filter.rs, and the harness_invariants / overhead_invariants suites.
• Completed the engine table (added Dragonfly, KiviDB, Chroma, Vertex AI).
• New "Traps that have actually cost time here" section. Every entry is a real incident from this repo's history, and every one of them looks fine while you are doing it.
• CLAUDE.md was a single line reading AGENTS.md. It now points there explicitly and carries the few things worth knowing if nothing else is read — one instruction set for both tools, so they cannot drift apart.

Verification

Every make target and every source path named in AGENTS.md was checked to exist on this branch, and make agent-check passes.

One judgement call worth flagging: I kept a single AGENTS.md with CLAUDE.md as a thin pointer rather than duplicating guidance. Two files saying almost the same thing is how instructions go stale. Easy to split later if Claude and Codex ever need divergent advice.

🤖 Generated with Claude Code

[github-actions]

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

Docs-plus-CI-plumbing, so items 1–6 of the usual invariant checklist (filter choke point, #151-4 per-config naming, corpus identity, start gates, destructive-suite ownership) aren't implicated — nothing here touches engine, dataset or measurement code. What is in scope is the thing 82738c0 set out to do: make make agent-check be the gate rather than resemble it. That framing is right, and the self-corrections in 5f4bfa4 / 54d9d06 (the make agent-check | tail -2 && git commit pipeline-exit-status incident, then moving the lesson out of the commit message into the docs where it gets read) are the standard this repo holds itself to — no point restating them.

The findings below are all the same shape as the one the PR already caught in itself: the new guard is a notch narrower than the claim it's named after.

Worth fixing before merge

1. The four new invariants don't run on the changes that break them. .github/workflows/ci.yml:14-22 (and :6-13 for pushes) filter on Cargo.toml, Cargo.lock, src/**, tests/**, datasets/datasets.json, experiments/configurations/**, .github/workflows/ci.yml. Makefile, AGENTS.md and CLAUDE.md are not in either list — and no other workflow picks them up (docker-build-pr.yml:7-14 doesn't either). So INV-A1/A2/A3/A3b are green-by-absence on exactly the PRs they exist for: a docs-only PR that renames a target reference in AGENTS.md, or a Makefile-only PR that renames agent-check-lint, never triggers CI, merges green, and — same filter on push — doesn't trigger CI on master either. Master then stays broken until the next src/** change, at which point the check job fails with a make: *** No rule to make target that points nowhere near the commit that caused it. This PR itself is exempt only because it happens to touch tests/** and ci.yml. Fix is three lines added to both paths: lists.

2. Nothing asserts what agent-check-lint / agent-check-test actually run. INV-A1 checks the direction CI → Makefile (check_job.contains("make agent-check-lint"), test_job.contains("make agent-check-test")) and INV-A2 only reads target names from the left of the colon. No invariant reads the recipe bodies. Before this PR the suite list lived in ci.yml:61-84, where deleting a suite showed up as a removed, named job step in the workflow diff. After it, deleting the cargo test --test overhead_invariants --release line from agent-check-test leaves both CI and the local gate green while that suite stops running entirely — which is tests/overhead_invariants.rs has never run in CI — INV-2/INV-3 are decorative, and #263's INV-4/INV-4b would land unfailable #265 verbatim (overhead_invariants existed and had never actually run in CI). Cheap to close in the same text-test style: assert agent-check-lint's recipe names cargo fmt --check and cargo clippy --all-targets -- -D warnings, and agent-check-test's names --lib --bins --release plus all three of integration_cli / overhead_invariants / harness_invariants. Mutation to verify: drop one suite line, expect red.

3. AGENTS.md's "Other Make targets" block describes make test wrongly, in both halves. The doc says make test # Unit tests only, debug profile (no Docker). Makefile:104-106 is cargo test --lib --release -- --nocapture — release, not debug, and --lib only. The missing --bins is the part that matters: this PR's own ci.yml comment (retained at the new "Unit + invariant suites" step) says the bulk of the unit tests — engine filter builders, CLI/config parsing, per-engine helpers — live in the binary crate and --lib alone never executed them. As written, an agent that edits a filter builder under src/bin/vector_db_benchmark/engine/ and runs make test runs none of its tests, and the doc tells them the only difference from CI is the profile. Either correct the description or make test --lib --bins.

4. INV-A1's banned-command scan only sees single-line run: steps. The loop is if !line.trim().starts_with("run:") { continue; }, so a block step is invisible:

   - name: Lint
     run: |
       cargo clippy --all-targets -- -D warnings

   That isn't a hypothetical form — ci.yml:46-48 already uses run: | for the audit step, in the very job INV-A1 guards. The guard's own docstring says "A raw command here is a second source of truth that agent-check cannot see", and the most natural way to reintroduce one slips past it. Scanning the whole job slice for the banned substrings (with cargo audit / cargo install cargo-audit explicitly allow-listed, as the comment already intends) is stricter and simpler than per-line run: detection. Mutation: add the three-line block above and expect red.

5. INV-A2 skips the brace form, which is where most of the targets are named. The target.ends_with('-') || matches!(next, Some('{') | Some('<')) bail-out is correct as far as it goes, but it means the three brace lines in AGENTS.md's integration-test block — {valkey,dragonfly,kividb,chroma}, {pgvector,qdrant,elasticsearch,opensearch}, {weaviate,milvus,mongodb} — contribute zero checked targets. 11 of the 15 integration-test-* targets the doc names go unverified, including three of the four this PR adds; rename integration-test-kividb tomorrow and INV-A2 stays green. This is the same near-vacuity the commit message describes finding in INV-A3, one file over. Expanding a brace set is a few lines, and it's the difference between the guard covering 4 target references and covering 15.

Optional / follow-up

• The agent-check comment block in the Makefile has two paragraphs duplicated verbatim (the --release is not incidental paragraph and the Docker-backed engine suites are NOT included line each appear twice), evidently from layering 82738c0 onto the original block. More substantively, it cites cargo fmt --check (ci.yml "Check formatting") and cargo clippy ... (ci.yml "Run clippy") — step names this same diff deletes in favour of Format + clippy (make agent-check-lint). Stale cross-references in a PR whose thesis is that stale docs mislead agents, and neither INV-A2 nor INV-A3 can see it because the Makefile isn't among the files those tests scan for doc claims.
• The "Match CI's Rust toolchain" bullet attributes to toolchain drift something that's mostly a flag difference: lint (Makefile:258-260) is a bare cargo clippy — no --all-targets, no -D warnings — so make check is clean on warnings by construction on any toolchain, not just an older one. Worth saying that directly, and pointing at agent-check-lint (or the existing check-strict) as the thing that actually matches CI.
• Workflow Rule 3's justification is garbled: "The one regression in the regression this rule was written for was selection logic that ended up in a binary…". The rule is good; the sentence needs a pass, and this is a file whose whole purpose is being read literally by something that can't ask what it meant.
• INV-A3b got a checked >= 15 coverage floor after the no-op-mutation lesson; INV-A3 has none. Given the structure block writes bare filenames, A3 currently validates about one qualified path (datasets/datasets.json) and would silently validate zero if that line changed. A floor of 1–2 there costs nothing and makes the near-vacuity assert itself instead of being remembered.
• The four new integration targets diverge slightly from the other nine (Makefile:128-214): no @echo "=== Running … ===" line and no --nocapture. Cosmetic, but the nine that exist are uniform and the log shape is what you read when one of them fails in CI.
• Not a finding, verified rather than assumed: make integration-test-vertex is safe to leave in the target list — tests/integration_vertex.rs:40-53 self-skips with a printed notice unless VERTEX_PROJECT is set, so an accidental invocation with ambient cloud credentials is a no-op rather than a live-project mutation. The ports the three new Docker targets export (6385 / 6386 / 8003) match ci.yml:312, :333, :354 exactly, and all four --test targets exist under tests/.
• The rationale recorded in 9dc0f13 for keeping the gate free of any model call — deterministic, free, offline-runnable, with model review left to the advisory PR workflow where a flaky answer blocks nobody — is worth keeping exactly where it is, next to the target someone would be tempted to edit.

Nothing in the PR body or diff tries to redirect a reader/agent away from reviewing it; the 🤖 Generated with Claude Code footer and the Co-Authored-By trailers are this repo's normal pattern.

Net: the direction is right and the CI-invokes-the-Makefile move removes a real drift surface, but four of the five findings above are the new guards being a notch narrower than their own names — and #1 is the one that matters most, because it means a Makefile- or AGENTS.md-only follow-up can break all four invariants with CI never running. #1 and #2 look like a handful of lines each; #3 is a one-line doc correction. Whether the brace-expansion and block-run: widenings (#4, #5) land here or as a follow-up issue is a judgement call, and the merge decision is a human maintainer's.

[fcostaoliveira]

The one red check here is inherited from master, not caused by this PR.

Integration tests (Milvus) fails with:

milvus-minio Error pull access denied for minio/minio, repository does not exist
or may require 'docker login': denied: requested access to the resource is denied

MinIO withdrew that release tag from Docker Hub, so tests/docker-compose.test.yml on master points at an image that no longer resolves and the job dies before a single test runs. Any branch cut from master inherits it. Fix is in flight as its own PR.

Everything this PR actually touches is green — including the two jobs it rewires, which now run make agent-check-lint and make agent-check-test instead of their own copies of the commands. That is the first live confirmation that the Makefile targets are the CI gate rather than a description of it.