Skip to content

The inspection depth field records the verdict, not the depth — and is false every time it is set #8

Description

@krapcys1-maker

README_READ is not a depth. It is the verdict "not verified" wearing a depth's name, and it is false in 100% of the cases it is assigned.

Where

src/lib/sources/code-inspection.ts:

depth: verified ? (hasEvalArtifacts ? "BENCHMARK_FOUND" : "CODE_INSPECTED") : "README_READ",

That README_READ branch is reachable only when examined.length > 0 — an inspection that opened no files returns INSPECTION_FAILED instead. So README_READ always means "we opened and read source files and found no qualifying match", and never means "we only read the README".

Why it matters

depth is the single field every downstream consumer keys on:

  • claim-ledger.tsinspectionDepthByRepo()
  • source-coverage.tsDEPTH_EVIDENCE_CEILING, and now the check that a required CODE_VERIFIED was actually obtained
  • scripts/audit-run.ts

Observed

Run cmtg7821u inspected three repositories and read 36 source files between them — Iron-mem opened 12 of 48 including src/bench.rs, synapse read src/synapse/encoding.py. All three stored "depth": "README_READ".

Everything downstream, every human reading the record, and the report itself then treated those as repositories whose code had not been read. This misled the maintainer directly: the state was reported to a user as "all three stopped at README_READ, none reached CODE_INSPECTED", which is the opposite of what happened.

It is this project's recurring defect in its purest form — a search that ran and returned nothing, stored as a search that never ran.

Suggested fix

Split the field:

  • depthReached — derived from what was actually opened. examined.length > 0 means at least code-inspection-level looking; eval artifacts mean benchmark level.
  • verdictCODE_VERIFIED | CODE_NOT_VERIFIED.

Promotion in maxEvidenceClassForSource then requires both (depthReached >= CODE_INSPECTED && verdict === "CODE_VERIFIED"), which preserves today's ceiling exactly while making the depth field true. Rename README_READ to something it can honestly mean, or drop it from the ladder.

Good first issue?

Yes, with care. The change is small and local, but every consumer of depth must be updated in the same commit, and each has a test.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    evidence-inflationA claim was graded higher than its evidence supportshelp wantedWe would genuinely like outside help on this onesilent-failureA failure that recorded itself as a success

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions