agmind-lab — evidence archive

The raw evidence behind every number published at agmind.ai.

Each directory under runs/ is one measurement run's evidence bundle, exactly as the harness sealed it:

runs/<run-id>/
  manifest.json     cell identity: system, runtime image tag, model path,
                    workload@revision, operating settings, launch command.
                    Runtime image DIGESTS and model artifact SHA-256 are
                    pinned in catalog/runtimes/ and catalog/models/ — the
                    bundle names them by catalog id.
  requests.jsonl    one record per request, including failures
  quality.jsonl     per-request quality-gate outcomes
  metrics.json      aggregates computed by the harness
  server.log        runtime output for the run
  sha256sums.txt    checksums of the files above

catalog/ is the registry that ties bundles to published numbers: systems, runtimes, models, workloads (with frozen corpus hashes), runs, claims and tested-configuration cards. Every claim names its run IDs and a DuckDB SQL query; re-running the query against the named runs' requests.jsonl or quality.jsonl must reproduce the published value exactly. The site's CI fails when it does not.

Headline results (from the sealed evidence)

Measured on AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151, Strix Halo), llama.cpp pinned by image digest, Qwen3.6-35B-A3B pinned by artifact hash — every number re-derived from raw run records in CI:

Question	Measured answer
llama.cpp Vulkan vs ROCm decode	15.9 vs 18.7 ms/token (claim)
Prompt cache: 2nd question over a 32k doc	860 ms vs 33,728 ms cache off (claim)
Reasoning on vs off, time to first answer token	20,191 ms vs 210 ms at equal task success (claim)
Concurrency ladder c1/c4/c8, first answer token	210 / 338 / 865 ms, 100% completion (report)

Full table of all 40 claims: RESULTS.md · machine-readable: claims.json

Verify a number yourself

The published values themselves are snapshotted in claims.json (same content as https://agmind.ai/claims.json), so verification closes offline: derive, then compare against the snapshot. In the SQL files, the $runs placeholder stands for the DuckDB read_json_auto([...]) list of the claim's run records — requests.jsonl per run, or quality.jsonl when the claim's query declares "source": "quality".

# pick a claim, e.g. catalog/claims/strix.qwen36.docsession.c1.ttft-q2-32k-cache.json
# it names run_ids and a sql file; run the SQL over those runs' records:
duckdb -c "SELECT median(ttft_ms) FROM read_json_auto(['runs/run-20260803-ds-cache-a/requests.jsonl', 'runs/run-20260803-ds-cache-a-r2/requests.jsonl', 'runs/run-20260803-ds-cache-a-r3/requests.jsonl']) WHERE ok AND item_id='en-32k-q2'"

Honesty notes

• Invalid and failed runs are preserved, marked with their reason — never deleted. A rerun gets a new run ID.
• Checksums prove a bundle has not changed since publication here. They do not prove the operator ran the test honestly — that is what the open harness and external reproduction are for: agmind-bench.
• Manifests record the exact launch commands, including local filesystem paths on the lab machines (node-b bundles embed its historical /home/beelinknode2/agmind-models mount; later runs use the neutral /var/lib/agmind/models). That is provenance, not configuration advice — sealed bundles are never edited retroactively, checksums included.

License

Data and metadata: CC BY 4.0. Attribute as "AGmind Systems Lab, agmind.ai".