Skip to content

README: restore the explicit 'Yes' answer in Does it work? #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
max-friedman merged 1 commit into from
Jul 28, 2026
Merged

README: restore the explicit 'Yes' answer in Does it work? #18

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 18 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,24 @@ weakened.

## Does it work?

Two data points, not one.

**A benchmark, adversarially:** run against
[TactBench](https://github.com/max-friedman/tactbench), whose README claimed
immunity to a specific failure mode, unmeasured. Round 1's job was to build a
probe for exactly that failure, and it scored **93.5% against a 50% floor** —
the claim had been false since the first commit, caught by the process rather
than avoided by it. Full account, round by round:
[`docs/CASE_STUDY.md`](docs/CASE_STUDY.md).

**A production app, at scale:** an informal predecessor of this protocol ran
independently for **120+ rounds** on a live consumer app, before that project
adopted the released version of this one — real shipped fixes, real security
hardening, and the actual source of
[proposal #003](proposals/003-roast-findings-need-verification.md): the
ground-truth verification step in `§E` exists because that app's real history
kept surfacing complaints that cited something genuinely observed and still
weren't real bugs — misdiagnosed causes and test-environment noise alike.
**Yes — two data points, both real:**

- **It caught a false claim a benchmark had shipped since its first commit.**
[TactBench](https://github.com/max-friedman/tactbench)'s README claimed
immunity to a specific failure mode — never actually measured. Round 1's
assignment was to build a probe for exactly that failure. It scored **93.5%
against a 50% floor**: the claim was false, and this process is what caught
it, not a human auditing the repo. Full account, round by round:
[`docs/CASE_STUDY.md`](docs/CASE_STUDY.md).

- **It holds up at real scale, not just on a benchmark.** An informal
predecessor of this protocol ran independently for **120+ rounds** on a live
consumer app — real shipped fixes, real security hardening — before that
project adopted this repo's released version. Its history is the actual
source of [proposal #003](proposals/003-roast-findings-need-verification.md),
already merged: the ground-truth verification step in `§E` exists because of
real complaints that history kept surfacing — genuinely observed, still not
bugs — that a naive check would have missed.

## Using it

Expand Down
Loading