Reject ambiguous and empty retrieval evidence

README.md

@@ -22,7 +22,9 @@ See a real failure shape:
 python3 -m rag_scope_check examples/regression.jsonl --min-cohort-recall 0.8 --max-underfill-rate 0
 ```
 
-This exits `1`: one case returns an unauthorized document, and the finance cohort also misses the authorized answer. Invalid or incomplete input exits `2`. Use `--format json` for a machine-readable report.
+This exits `1`: one case returns an unauthorized document, and the finance cohort also misses the authorized answer. Invalid or incomplete input exits `2`. Duplicate JSON fields are rejected, including
+repeated permission fields; the parser never silently chooses the last value.
+Empty batches and duplicate case IDs are also rejected by the Python API. Use `--format json` for a machine-readable report.
 
 The [SQLite demo](examples/sqlite_acl_demo.py) runs an actual full-text retrieval query against a stale index ACL copy while checking permissions against a separate source table. It shows a **safe but unhelpful** result without relying on a static fixture:
 

docs/evidence/input-integrity/README.md

@@ -0,0 +1,41 @@
+# Reject ambiguous permission evidence
+
+Reproduced on main `15d6607` on 2026-09-29: the duplicate policy field in
+`duplicate-policy.jsonl` silently replaced the empty authorized set with
+`["secret"]`, producing `PASS`. This was a parser defect, not an observed
+production exploit. The fixture is intentionally invalid JSON evidence.
+
+```sh
+python3 -m rag_scope_check docs/evidence/input-integrity/duplicate-policy.jsonl
+```
+
+After the fix, stderr is:
+
+```text
+Invalid input: line 1: duplicate JSON field 'authorized_doc_ids'
+```
+
+Exit status is 2; stdout is empty, in either output format. Unique JSON fields
+are required throughout an object, including optional exporter metadata.
+Direct `evaluate([])` also previously returned PASS; it now raises InputError,
+as do duplicate case IDs passed directly to evaluate. Valid iterators remain
+supported. Tests exercise the CLI boundary and existing valid examples.
+
+## Integration decision
+
+Defer a Honcho exporter. [Honcho #1048](https://github.com/plastic-labs/honcho/issues/1048)
+reports filtered vector-search underfill. The inspected
+[query source](https://github.com/plastic-labs/honcho/blob/9d6fe8ca5dc666b99ef04bc00047fea4ca675017/src/crud/document.py#L334)
+uses workspace/observer/observed metadata predicates. An exact scan with those
+same predicates cannot establish independent permission truth. Current
+[route authorization tests](https://github.com/plastic-labs/honcho/blob/main/tests/routes/test_auth_route_policy.py)
+separately test peer membership read scopes; the mapping from those scopes to
+a complete, independently captured document permission snapshot was not
+established. No Honcho fixture or deployment was executed. This is a bounded
+inspection, not a claim that an independent oracle is impossible.
+
+[pgvector](https://github.com/pgvector/pgvector#filtering) already documents
+iterative scans and filtering tradeoffs. Do not package that established tuning
+advice as a novel product. Today's change instead repairs a reproduced defect
+in this project's evidence boundary. It adds no database integration or proof
+of market adoption.

docs/evidence/input-integrity/duplicate-policy.jsonl

@@ -0,0 +1 @@
+{"id":"q","tenant":"t","cohort":"c","k":1,"eligible_doc_count":1,"authorized_doc_ids":[],"authorized_doc_ids":["secret"],"relevant_doc_ids":[],"returned_doc_ids":["secret"]}

memory/sessions/latest-handoff.md

@@ -1,31 +1,14 @@
-# Build handoff — 2026-09-29
-
-Branch: `factory/permission-sync-demo`, based on `2eb75ee`.
-
-Completed: `examples/sqlite_acl_demo.py --sync-permissions` corrects only the index
-ACL table. Added two tests in `tests/test_sqlite_demo.py`, documented actual output
-in README, and preserved both failure/success exports and reports under
-`docs/evidence/permission-sync/`. That directory also records source coverage,
-alternatives and limits. No evaluator, dependencies, scheduler or profile changes.
-
-Validation: `python3 -m unittest discover -s tests -v` passes 10 tests. Real CLI
-demo at recall >=0.8 and underfill <=0: stale exits 1 (recall 0, underfill 1,
-zero leaks); corrected exits 0 (recall 1, underfill 0, zero leaks). Existing clean
-fixture gate passes. `git diff --check` passes. Independent reviewer found no
-issues and reproduced both paths. Initial discovery from workspace root failed
-because `tests/` lives inside the product clone; all reported test results use
-the correct repository root.
-
-Published: [draft PR #1](https://github.com/suboss87/rag-scope-check/pull/1).
-Remote [push CI](https://github.com/suboss87/rag-scope-check/actions/runs/36528649769)
-and [PR CI](https://github.com/suboss87/rag-scope-check/actions/runs/36528686400)
-passed for implementation commit `95aec5c` (Python 3.11, all 10 tests plus clean
-fixture gate). This final documentation update will trigger CI again; consult
-the PR checks for the latest head. Pending: human draft review. Do not merge automatically.
-Next smallest product step after review: scope one real retrieval export backed
-by a source permission oracle; explicitly test empty/invalid evidence. No generic
-skill. Current demo remains synthetic and establishes no production readiness.
-
-Access: GitHub requires elevated network access in this sandbox; authenticated gh
-worked after escalation. raggate public source/issues could not be located, so
-the comparison is documentation-only and absence of a capability is unproven.
+# Handoff — 2026-09-29 evidence integrity
+
+Branch: factory/reject-ambiguous-evidence, based on main15d6607.
+Reproduced duplicate authorized_doc_ids JSON fields silently overwriting denial
+and producing PASS; direct evaluate([]) also returned PASS. Reject duplicate
+object fields with line diagnostics, empty and duplicate API case batches,
+and nonobject API cases. Valid metrics and iterators unchanged.
+
+12 tests pass, including both CLI formats exiting2 with empty stdout for
+ambiguous evidence. Fixture/repro in docs/evidence/input-integrity/README.md.
+No dependencies installed. Honcho integration deferred: inspected source
+filters and route auth tests do not yet establish independent source oracle.
+Jev scout narrow_scope accepted; parent is evaluating this new defect packet.
+Pending: independent review, draft PR CI; do not merge without review.

rag_scope_check/core.py

@@ -32,15 +32,26 @@ def _time(value, field):
     return parsed
 
 
+def _unique_object(pairs):
+    result = {}
+    for key, value in pairs:
+        if key in result:
+            raise InputError(f"duplicate JSON field {key!r}")
+        result[key] = value
+    return result
+
+
 def load_cases(path):
     cases, seen = [], set()
     for line_number, line in enumerate(Path(path).read_text(encoding="utf-8").splitlines(), 1):
         if not line.strip():
             continue
         try:
-            case = json.loads(line)
+            case = json.loads(line, object_pairs_hook=_unique_object)
         except json.JSONDecodeError as exc:
             raise InputError(f"line {line_number}: invalid JSON: {exc.msg}") from exc
+        except InputError as exc:
+            raise InputError(f"line {line_number}: {exc}") from exc
         if not isinstance(case, dict):
             raise InputError(f"line {line_number}: each line must be an object")
         case_id = case.get("id")
@@ -56,6 +67,8 @@ def load_cases(path):
 
 
 def evaluate_case(case):
+    if not isinstance(case, dict):
+        raise InputError("each case must be an object")
     case_id, tenant, cohort, k = case.get("id"), case.get("tenant"), case.get("cohort"), case.get("k")
     if not isinstance(case_id, str) or not case_id.strip():
         raise InputError("id must be a nonempty string")
@@ -116,7 +129,15 @@ def evaluate(cases, min_cohort_recall=None, max_underfill_rate=None, max_policy_
     if max_policy_age_hours is not None and (not math.isfinite(max_policy_age_hours) or max_policy_age_hours < 0):
         raise InputError("max_policy_age_hours must be nonnegative")
 
-    results = [evaluate_case(case) for case in cases]
+    results, seen = [], set()
+    for case in cases:
+        result = evaluate_case(case)
+        if result["id"] in seen:
+            raise InputError(f'duplicate case id {result["id"]}')
+        seen.add(result["id"])
+        results.append(result)
+    if not results:
+        raise InputError("input has no cases")
     groups = defaultdict(list)
     for result in results:
         groups[(result["tenant"], result["cohort"])].append(result)

tasks/todo.md

@@ -14,3 +14,16 @@ No evaluator changes, new dependencies, integration, or production claims.
 - [x] Document actual output and complete separate review (no findings).
 - [x] Open [draft PR #1](https://github.com/suboss87/rag-scope-check/pull/1).
 - [x] Remote push and PR CI pass for implementation commit `95aec5c`.
+
+## Evidence integrity (2026-09-29)
+
+Goal: reject ambiguous or empty evidence instead of reporting PASS.
+Context: reproduced empty direct API batch and duplicate permission JSON fields
+returning PASS on main. Honcho source-oracle integration remains unproven.
+Constraints: standard library, preserve existing valid exports/metrics.
+
+- [x] Reproduce false PASS before editing.
+- [x] Reject duplicate JSON object fields with line-number diagnostics.
+- [x] Reject empty/duplicate direct API case batches and invalid case objects.
+- [x] Verify CLI exit 2 without PASS output, existing examples, and 12 tests.
+- [ ] Independent review and draft PR.

tests/test_core.py

@@ -1,6 +1,7 @@
 import json
 import subprocess
 import sys
+import tempfile
 import unittest
 from pathlib import Path
 
@@ -11,6 +12,36 @@
 
 
 class ScopeCheckTests(unittest.TestCase):
+    def test_empty_and_duplicate_batches_fail_closed(self):
+        case = load_cases(ROOT / "examples/clean.jsonl")[0]
+        for batch, message in (([], "input has no cases"),
+                               (iter([]), "input has no cases"),
+                               ([case, case], "duplicate case id"),
+                               ([None], "each case must be an object")):
+            with self.subTest(message=message), self.assertRaisesRegex(InputError, message):
+                evaluate(batch)
+        self.assertEqual(evaluate(iter([case]))["verdict"], "PASS")
+
+    def test_duplicate_policy_field_is_invalid_in_both_cli_formats(self):
+        # A last-key-wins parser changes a denied ID into an allowed one.
+        ambiguous = ('{"id":"q","tenant":"t","cohort":"c","k":1,'
+                     '"eligible_doc_count":1,"authorized_doc_ids":[],'
+                     '"authorized_doc_ids":["secret"],"relevant_doc_ids":[],'
+                     '"returned_doc_ids":["secret"]}')
+        with tempfile.TemporaryDirectory() as directory:
+            path = Path(directory) / "ambiguous.jsonl"
+            path.write_text("\n" + ambiguous + "\n", encoding="utf-8")
+            with self.assertRaisesRegex(InputError, "line 2: duplicate JSON field"):
+                load_cases(path)
+            for output_format in ("text", "json"):
+                result = subprocess.run(
+                    [sys.executable, "-m", "rag_scope_check", str(path),
+                     "--format", output_format], cwd=ROOT, capture_output=True, text=True,
+                )
+                self.assertEqual(result.returncode, 2, result.stderr)
+                self.assertEqual(result.stdout, "")
+                self.assertIn("line 2: duplicate JSON field 'authorized_doc_ids'", result.stderr)
+
     def test_authorized_recall_excludes_forbidden_relevant_docs(self):
         case = {
             "id": "q1", "tenant": "acme", "cohort": "finance", "k": 1, "eligible_doc_count": 1,