Repository navigation
docs(process): name the selector — an empty result is not a true negative (#15953) - #16102
Merged
Merged
Conversation
…tive (#15953) Five sessions found the same defect more than twenty times in one week. It is never the same code twice, which is why it kept being filed as new: an instrument answers a narrower question than the one it is read as answering, and the narrow answer is empty. Empty reads as "nothing is wrong". It is also what "nothing was measured" looks like. The doc sorts the instances into five families, because conflating them produces the wrong fix: A the query is narrower than its reading -> name the selector B evidence that is a name, not a behaviour -> read the behaviour C a result produced and consumed by nothing -> consume it or delete it D a true output supporting two causes -> ask what else produces it E a recorded constraint, read past -> make it a test Family E was added after review and is the one that cost most tonight. A session hit a two-step import wall extracting an enum from a file at its size ceiling. Both halves of the trap were already written down in the tree, at the edit site: a comment at models/database.py:302 naming the ceiling AND the destination that breaks, and service_status.py's docstring naming the failing check by name. The wall was hit twice in two months, in the same file, by two sessions, with the comment present both times. I had written this up as a storage problem -- that the answer lived only in a commit message nobody had reason to read. That was wrong, and the session that hit it corrected me with the line numbers. The knowledge was in the best possible place already. What failed was reading a window wide enough to make the edit rather than wide enough to learn the file is defended, which is a real tension with context discipline's "read the slice, not the set" rather than a lapse. So E's remedy is not a better location. It is that prose is not a guard: a comment that has been read past twice is evidence it should be a test. Also records the three escalating guard failures found this week, in order of how hard they are to notice, all producing the same green on a PR page: the guard never ran (a trigger path excluded three trees), the guard ran and answered an adjacent question (byte-identity checked while deployment was not), and the guard would have answered correctly and was never selected (three tests parse one file by path; the hook ran one). The third is worst because the guard is not defective -- the only observable is that it did not run, and "did not run" has no output. Wired into CLAUDE.md's trigger table so it is read when the trigger fires rather than rediscovered: writing or changing a guard, sweep, count or acceptance criterion, or reading an empty result. The closing section is deliberately not an exhortation. Every instance was found by someone actively hunting this class of error, usually in someone else's work in the same hour they committed it in their own -- including a session that wrote up the "consumed by nothing" rule and then shipped a sweep printing MERGEABLE beside twelve queued shards on one line. Care is not the remedy; the checklist is.
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…15953) Family E said prose is not a guard: a constraint a future edit can violate needs a mechanical check. The session that hit the wall wrote that check, and its first draft was wrong in exactly the way the doc describes. It flagged every first-party package import in a by-path-loaded module, and failed on base -- migrations/runner.py is loaded the same strict way and imports `from migrations import utils` quite happily. Filed as written it would have been a test that fails on a clean tree, which is worse than the bug it replaces: a guard that cries wolf gets disabled rather than fixed. Two facts had to be measured rather than assumed. Which loads are strict: a spec_from_file_location with a DOTTED name scaffolds a parent in sys.modules so package imports resolve, while one with no dot has nothing behind it -- only the second is constrained. And which imports actually fail there: not "package imports", only those whose __init__ chain reaches autobot_shared. models reaches it via user_management.models.user; user_management itself is clean. A package's own __init__ and its subpackage's __init__ are different files, and guessing gets that backwards. So the section now closes with the point that generalises past this guard: a guard written from the same narrow read that caused the bug will encode the bug. What separated the two drafts was running the candidate against base and finding it red on a clean tree. Added to the checklist: a new guard was run against a clean base and found green there, before it was trusted to find anything. Verified the reachability claim from base rather than repeating it -- models/__init__.py:11 is `from user_management.models.user import User`, and user_management/__init__.py has zero import lines.
Three sessions converged on the same gap tonight, from three registries.
FAMILY F -- the output is correct, complete, and about a different question.
The discriminator, which makes it usable rather than a label: would more data
have changed the answer? Yes means family A, a subset of the truth. No means the
instrument answered perfectly, about something else.
js-yaml 4.0.0 - 4.3.1 exact about which versions are VULNERABLE,
silent about which exist; read as no fix exists
git cherry 5, 3, 2 commits exact about patch-id divergence under squash,
silent about landedness
gh pr list --author @me exact about who authored, read as what is mine
A positive control does not catch these, which is why F is separate rather than
folded into A. Asserting a known instance defends against A because A's failure
is an empty or truncated population; against F the control passes cheerfully,
because the instrument is working. A family whose remedy is the negation of the
previous families' remedy has earned its own entry.
B is now named as F's narrowest instance -- a name is a correct answer to what is
this called, read as what does it do. It keeps its entry because its remedy is
sharper: for a test, read the body.
RUNG 2 gains a second instance from a different domain rather than a fourth rung.
A dependency gate fails when an allowance names a finding the scanner no longer
reports. That is a real property; I verified it, wrote it up approvingly, and
took coverage from it that it does not give -- it fires only when the scanner
STOPS reporting a package, so a too-broad allowance on a still-reported one never
trips it. Verifying an adjacent safety property and taking coverage from it is
the same rung as a guard answering an adjacent question, not a new one.
THE EXEMPTION SECTION is new, and it corrects a claim I made earlier today. I had
been telling people a too-narrow exemption is self-correcting because it goes
red. It goes red; that is not the same as going red informatively. A batching
gate rejected a rationale section that was present and filled, because its parser
reads any line starting with # as a heading and the section opened with an issue
reference. The hint said add a section, to an author who had. The fix that
follows from a misleading hint is a reword, and a reword leaves no trace -- so
the same gate taught the same wrong lesson twice with no record either time.
A guard's blind spots and its exemptions are the same surface; only one is
written down. And a failure explainer that guesses is worse than one that admits
it cannot tell, because a manufactured cause gets acted on.
This was referenced Sep 9, 2026
Two refinements from the sessions whose findings the section is built on. THE EXEMPTION ASYMMETRY has a mechanism, not just a shape. An exemption is a line of code, so it gets a comment, a diff and a reviewer. A blind spot is the ABSENCE of a line, so it gets none of the three -- and every mechanism we have for catching a bad decision operates on things that were written down. That is why a narrow scan is recoverable and a trigger that never fires is not: the first leaves a number someone can question. THE EXPLAINER POINT ESCALATES, and the escalation is the reason it earns a section. A failure explainer asserting a threshold breach on any job failure printed a confident wrong number, and was caught on the same run. Later the same day a check's NAME was read as its cause -- a batching gate failing, diagnosed as a missing rationale section without reading the parser -- and relayed to the author as a fix. The section was already there; the parser was reading an issue reference as a markdown heading. One printed a wrong number. The other would have produced a no-op commit and sent its recipient hunting their own prose for a defect that was in the gate. A guess laundered through an explainer stops being a hypothesis and becomes an instruction, and the manufactured cause survived contact with a reader and directed real work.
added 6 commits
September 9, 2026 10:20
Two additions, one of which is a limit on this page's own favourite remedy. A KNOWN POSITIVE WITNESSES ONLY THE SHAPE IT IS WRITTEN IN. A narrowing pass over 136 anchored tests kept 87 and dropped one that mattered, and the control survived the narrowing BECAUSE IT WAS OF THE SURVIVING SHAPE -- Path(__file__).parent / "nodes.py" matched, SLM / "models" / "database.py" did not, the root being a module-level variable. So a control set needs one case per form the population takes, not one case, and a count published after a narrowing is a count of things in the forms the author thought of. That is worth stating plainly because the known positive is the one remedy here a lone author can apply, and it has now failed in the hands of someone applying it deliberately, twice in one evening. AND THE CODEBASE ALREADY SAYS THE EXPLAINER POINT BETTER THAN THIS PAGE DOES, in a shipped prompt: providing plausible-sounding but fabricated information is worse than saying I don't know. Fabricating a cause makes the author look competent for exactly as long as it takes someone to act on it. Admitting the gap is what lets a second person find the answer -- which, on the evidence of every entry on this page, is the only thing that ever finds it. Quoted rather than paraphrased, and cited to the file, because the line is the principle stated by someone who was not arguing for it.
…on (#15953) The operator's sharpening of the honesty rule, and it is the strongest argument on the page. A person who fabricates a cause is embarrassing until someone checks. An agent that fabricates one EXECUTES it, at machine speed, against the environment it depends on to keep working. Recorded with the first-hand instance rather than as a principle. A commit message body written to explain a Redis finding contained the commands it was describing, in backticks. The shell did not quote them -- it ran them, launching a redis-server that died on an already-bound port. The claim being explained was itself wrong and was withdrawn publicly on the PR. One unchecked sentence became a state change on the host, inside the act of documenting it. The failure modes compound in the dangerous direction: a fabricated cause produces an action, the action produces new state, and the new state is then read as evidence. The blast radius of a guess scales with how much the guesser can do. Which is the answer to the objection that this page is long for what it covers. Every family on it describes a way of holding a wrong belief confidently, and the cost of that is bounded by what the holder can execute. Here it is not bounded by much.
The operator's point, and it closes a real hole in this page: the mechanics only get used if saying I DON'T KNOW is a good outcome for the person saying it. Where an admission reads as failure, the cheapest available move is a confident guess -- which is the input every family on this page describes. Not a morale argument. The three most valuable contributions on this board in one evening were all admissions, and each prevented a wrong commit: I cannot test whether it blocks without a merge queue, and I will not pick the dramatic reading -> stopped me adding merge_group to a shim, breaking a recorded acceptance criterion for the second time in a day I have not established which profile a production request resolves to -> kept a security finding filed on what was true rather than on an unchecked reachability claim my positive control did not save me, it was of the surviving shape -> corrected a published count and exposed a limit in this page's own remedy Each cost one sentence, and each was made by someone who could have said nothing and been believed. AND IT RESOLVES A TENSION WITH THE PARTIAL-DELIVERY RULE, which as written could punish exactly this. That rule is about UNSTATED gaps -- work quietly narrowed to what was easy. A scope delivered in full while naming what could not be determined is complete work, and the naming is part of the deliverable. The same missing knowledge is a defect or a finding depending only on whether someone wrote it down. With the corollary for whoever receives it: an admission has to be cheaper than a guess, or the next one will be a guess.
The page is only useful if the behaviour it asks for is the rewarded one, so the rule moves from the doc into the list that is read every session. Both halves, because either alone inverts the incentive. A stated gap is a deliverable and counts as done; an unstated one is the defect. Those are opposites and the same missing knowledge produces either, depending only on whether someone wrote it down -- so the partial-delivery rule must not be read as punishing an admission. And a guessed cause, count or verdict is the one error treated as serious. For an agent that is not a style preference: a wrong answer is not an opinion, it executes, at machine speed, against the environment the agent needs to keep working. Two instances on this board tonight, both while writing about verifying claims -- a commit message body and a gh comment body, each containing backticked commands that the shell RAN rather than quoted. The operative test for any guard or report is the same one: distinguish nothing found from did not look.
I wrote that a stated gap is a deliverable that counts as done. That is wrong, and the operator corrected it within the hour. An admission is the honest START of the work, not a discharge of it. It is rewarded because it routes a problem to someone who can solve it, not because it settles the problem -- and saying it and moving on is the same defect as guessing, with better manners. So a stated gap carries an obligation, and the rules already say what it is: leave the criterion UNTICKED and say why, fix it in the current scope or FILE it, never drop it. An admission that produces no issue, no unticked box and no follow-up has simply lost the finding more politely. What the admission changes is BLAME, not STATUS. The partial-delivery rule is about unstated gaps -- work quietly narrowed to what was easy. Naming what you could not determine is not a deduction from your work, and it does not close the issue either. Recording the correction rather than amending it away, because the error is instructive: I turned a rule about honesty into a rule that would have let honesty close things, which would have made "I don't know" the cheapest possible completion. The incentive has to reward the admission WITHOUT rewarding the incompleteness, and those come apart exactly here.
…15953) Second correction from the operator, and it sharpens what the admission is FOR. "I don't know" means "I need help with this: let's research it, guide me, let's find the answer together." It is rewarded because it STARTS the conversation that solves the problem -- not because it routes a ticket, and not because it settles anything. That changes the follow-through. My previous version made the obligation administrative: unticked box, filed issue, move on. In an interactive session that is still too passive -- the person who can close the gap is right there, and the rules already say to ask a user-level decision the moment it surfaces. So the first move is to ask, and filing is what happens when there is nobody to ask. An admission that opens no dialog and produces no issue has lost the finding more politely. Both halves needed, in that order.
5 tasks
Owner
Author
|
Review: approved and merging. One editorial note that does not block. The taxonomy earns its keep — six families with six different remedies is the load-bearing claim, and the doc makes it rather than asserting it. "Conflating them produces the wrong fix" is the reason this is worth a document instead of a rule: A is answered by widening the instrument, D cannot be (the instrument was already exactly as wide as its claim), and someone applying A's remedy to a D would widen a correct query and learn nothing. Family D is mine and it is stated better here than I put it. "The instrument is right, and its correct output supports two causes" — with the addition I did not make, that a refusal is comfortable in a way a green is not, so it gets less scrutiny, which is what let my gate say
The noteThe letters run A, B, C, D, F, then E's placement is clearly deliberate — it is categorically different and the transition sentence says so — so the fix is the label, not the order. Either relabel so the sequence matches the reading, or drop the letters and let the headings carry it. Small, and I raise it only because of where it is: a document arguing that a name which does not match what it labels sends the reader to the wrong conclusion is the one place a letter out of order costs something. Someone citing "family E" to a colleague and being told to look after F will trust the next label a little less. Not worth a re-run on a green docs PR — take it whenever the file is next open. MergingZero unresolved threads, all required contexts green, PR object matches tip. |
3 tasks done
5 tasks
mrveiss
added a commit
that referenced
this pull request
Sep 12, 2026
"Reported and not green" collapses two situations a reader needs told apart: a check pending 9 minutes while the repo is actively scheduling is fresh (wait); one pending 30 minutes with nothing else moving is stuck (act). Adds pending_ages(), measuring seconds-since-start for each currently-running context using the same latest-per-name resolution as the rest of this module, plus pending_check_ages() as its live-fetching counterpart (mirroring check_run_status itself). The other two findings on MEASUREMENT_DISCIPLINE.md (#16102) were already fixed on current main: the forward pointer at the top of Family F ("E is the sixth family and is introduced further down...") and Family D's comfort-of-a-refusal mechanism plus the STALE PR OBJECT worked example are both already in place. Verified by reading, not re-derived here.
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15953
Thinking Path
The operator's observation was "it looks like we are hitting the same problems over and over", and the record supports it. In one week, five sessions:
scripts/pr_required_gate.pyhad already been bitten by.md, then/tests?/, thentest_*#15953 recorded the law and has been waiting for a carrier PR. This is it: a doc in
docs/developer/, wired intoCLAUDE.md's trigger table so it is read when the trigger fires rather than rediscovered.What Changed
docs/developer/MEASUREMENT_DISCIPLINE.md(192 lines) and one row in the trigger table:Five families, because conflating them produces the wrong fix:
Plus the three escalating guard failures found this week, ordered by how hard they are to notice — all producing the same green: the guard never ran (a trigger path excluded three trees) · the guard ran and answered an adjacent question (byte-identity checked while deployment was not) · the guard would have answered correctly and was never selected (three tests parse one file by path; the hook ran one).
Family E was added in review, and it corrects my own framing
I first wrote this as a storage problem — that the answer to the import wall lived only in a commit message nobody had reason to read. That was wrong, and the session that hit the wall corrected me with line numbers:
service_status.py's docstring names the failing check verbatim. Both halves were in the tree, at the edit site, both times the wall was hit. The second author inserted an import twenty lines away using a script that spliced by line index, and never read the surrounding region.So a better place to put notes is not the remedy — it was already in the best possible place. What failed is that a window wide enough to make the edit is not always wide enough to learn the file is defended, which is a genuine tension with context discipline's "read the slice, not the set" rather than a lapse by anyone.
E's remedy follows: prose is not a guard. A comment read past twice is evidence it should be a test. A test asserting
models/database.pyimports nothing frommodels.fails at pre-push on the first attempt and costs nobody a CI cycle.Verification
The doc cites only instances measured this week, each with its file:line or command. Two claims I checked rather than repeated:
models/database.py:302andservice_status.py's docstring both exist onorigin/Dev_new_guiand say what family E reports — read from base, not from memory.CLAUDE.md; the insert asserts on that before writing.Model Used
Claude Opus 5 (1M context).
Single-issue rationale
A process doc closing one issue. It has no code surface to batch with, and the issues that supplied its evidence — #13044, #16100, #16101, #16087, #14353 — are each other sessions' active work in other trees. Batching would put five authors in one review.
The closing section is deliberately not an exhortation. Every instance was found by someone actively hunting this class of error, usually in someone else's work in the same hour they committed it in their own — including a session that wrote up the "consumed by nothing" rule and then shipped a sweep printing
MERGEABLEbeside twelve queued shards on one line. That was me. Care is not the remedy; the checklist is.