bug(stories): reject semantically incomplete generated contracts before overwrite

[Serhan-Asad]

Problem

PDD accepts and writes an AI-generated Story contract when it contains the
required Markdown headings and no placeholder tokens, even if it omits
behavior-changing obligations from the human Story or original issue.

This is operationally significant because the generated contract is combined
with the human Story and becomes the oracle used by pdd detect --stories and
Story-aware fixes. An omitted obligation can therefore let a prompt or
generated-code regression pass.

The generation prompt already tells the model to preserve distinctions
precisely. The missing behavior is an acceptance gate that proves it did so.

Reproduction

An anonymized real-world PDD 0.0.309 run used:

• an approved one-sentence human Story;
• a canonical issue body refined to state six contract distinctions explicitly;
• sync_user_story_contract() with the canonical issue URL;
• force=True and temperature=0;
• progressively increased strength/time, ending at 1.0/1.0; and
• independent high-effort review after each generated result.

Every generated result was structurally valid. After multiple supported
regenerations, the final contract still omitted or compressed four separately
testable source obligations:

1. recoverable HTTP 401 and 403 as explicit branches distinct from terminal
   authorization failure;
2. the complete loading/error/reconnecting/busy/inert no-latch state set for a
   workflow stage;
3. explicit stale/context-block and late-result-contamination observations; and
4. positive preservation of already-permitted test-auth recovery eligibility.

The final call returned changed=true, cost 0.046558, model
vertex_ai/gemini-3.1-pro-preview, and a structurally valid contract. PDD
therefore overwrote the contract and reported success.

This is a general failure shape: a one-pass free-form summarization can collapse
individually testable source clauses while remaining fluent and structurally
valid.

Expected

PDD should accept a generated Story contract only when every in-scope,
behavior-changing obligation from the human Story and issue is represented in
the contract without broadening the Story.

Equivalent wording and implementation-neutral paraphrases must remain valid.
Completeness must not depend on exact keyword matching.

If completeness cannot be established, PDD should fail before writing the new
contract and report which source obligations are missing, ambiguous, or
over-broadened.

Actual

_llm_generate_story_contract() checks only that:

• the response is non-empty;
• all required section heading strings are present; and
• no template placeholder token remains.

_generate_and_write_contract() then writes the result immediately.

A fluent contract can therefore omit issue obligations, overwrite a better
existing contract, and become the downstream regression oracle.

Root cause

The underlying defect is an acceptance-boundary gap in Story contract
generation.

The generation prompt expresses a semantic completeness requirement, but the
implementation reduces acceptance to Markdown shape. There is no source-clause
inventory, coverage mapping, or semantic review between model output and the
write.

This is not primarily a model-quality problem. Stronger settings or another
model may improve the probability of completeness, but cannot make a single
unconstrained generation call a completeness guarantee.

Detectability gap

Current tests stub contract generation with hand-authored, already-complete
contract bodies. They verify headings, generated headers, Story-hash sync,
selected fixture phrases, and downstream use of Story plus contract.

They do not test the critical failure mode:

A model returns a structurally valid contract that omits one explicit source
obligation.

Consequently, the suite can pass while the production writer accepts an
incomplete oracle.

Scope

Add a bounded semantic-completeness acceptance step to generated Story
contracts. A durable correction should:

• identify behavior-changing source clauses that are in scope for the human
  Story;
• give those clauses stable identities or anchors;
• require an explicit, machine-checkable mapping from each required clause to
  contract evidence;
• permit semantic paraphrase rather than exact wording;
• detect omissions and inappropriate broadening;
• fail before write when coverage is incomplete or indeterminate; and
• return actionable diagnostics naming uncovered source clauses.

Stable clause IDs may come from explicit issue/rule IDs where present or be
derived from stable source anchors. The exact storage format may be a contract
section or accompanying structured result; this issue does not require one
specific internal representation.

For initial Story generation, the existing human Story may still be written if
contract generation fails, but no semantically incomplete contract should be
created. For contract synchronization, the previous contract must remain
byte-for-byte unchanged when the replacement fails acceptance.

Acceptance criteria

• A contract with every required heading but one uncovered source
  obligation is rejected.
• Every in-scope behavior-changing Story/issue clause has a stable ID or
  source anchor and a machine-checkable coverage disposition.
• Equivalent observable paraphrases can satisfy a clause without copying
  its exact text.
• Story scope remains authoritative: issue detail may refine the Story but
  may not broaden it.
• Missing, ambiguous, conflicting, or over-broad coverage produces a
  non-success result with actionable clause-level diagnostics.
• sync_user_story_contract() does not overwrite the existing contract
  unless semantic acceptance succeeds.
• Initial Story generation does not write an incomplete contract when the
  semantic gate fails.
• Only an accepted generated contract can become the downstream
  pdd detect --stories oracle.
• Existing required-section, placeholder, header, Story-hash, and Candidate
  Prompt behavior remains covered.

Regression tests

Add deterministic tests that do not depend on live model wording:

1. Define a synthetic Story/issue fixture with stable obligations such as
   AUTH-RECOVER-401, AUTH-RECOVER-403,
   NO-LOADING-ERROR-RECONNECTING-BUSY-INERT-LATCH,
   NO-STALE-CONTEXT-BLOCK, NO-LATE-RESULT-CONTAMINATION, and
   PRESERVE-PERMITTED-TEST-AUTH.
2. Stub the generator with a contract containing every required Markdown
   section but omit one obligation. Assert rejection.
3. Repeat with fluent wording that preserves all obligations through the stable
   mapping. Assert acceptance without requiring exact source wording.
4. Exercise sync_user_story_contract(force=True) with an existing contract
   and an incomplete replacement. Assert changed is false, the old bytes and
   hash are unchanged, and diagnostics identify the missing clause IDs.
5. Exercise initial Story generation with the same incomplete response. Assert
   that the human Story may remain but no contract file is written.
6. Simulate an unavailable or indeterminate semantic reviewer. Assert
   fail-closed behavior rather than overwrite.
7. Prove downstream detection cannot receive the incomplete candidate as its
   oracle.

These tests should fail against current 0.0.309 behavior and remain stable
across model/provider wording changes.

Non-goals

• Requiring exact string copies from the issue.
• Treating every issue sentence, implementation note, or evidence detail as a
  mandatory contract clause.
• Making the issue override or broaden the human Story.
• Hand-editing generated contract files.
• Solving cross-dev-unit ownership/topology completeness tracked separately by
  Require story-to-dev-unit completeness before claiming regeneration coverage #2358.
• Replacing executable Story regression tests or the Story regression gate.
• Guaranteeing deterministic LLM prose.
• Implementing the broader clause-to-code trace system from feat: emit clause-to-code traces and gate incomplete PDD candidates #1476.

Evidence

Installed pdd-cli 0.0.309 and current upstream main have byte-identical
versions of the relevant prompt and implementation. Current main does not fix
the defect.

Relevant source behavior:

• generate_story_contract_LLM.prompt asks the model to preserve distinctions;
• _llm_generate_story_contract() validates headings and placeholder absence;
• _generate_and_write_contract() writes immediately after those checks; and
• _compose_story_oracle() later combines that contract with the Story as the
  validation oracle.

The affected workflow originated in PR #1500 and was integrated through PR
#1501. Private reproduction receipts include three independent high-effort
reviews and exact frozen hashes; they can be provided to maintainers without
publishing the private downstream repository.

Related work

Parent: #833

• creating generated user stories #1356 / PR feat(test): author user stories from the issue, independent of the prompt (#1356) #1360 - issue-derived Story authoring.
• PR feat(user-stories): human-verified Story + generated AI contract (durable, readable) #1500 / PR Epic: user-story overhaul  #1501 - introduced the two-file generated-contract workflow.
• Goal: E2E verification that user-story regression oracles (EPIC #1816) are complete, useful, and merge-ready #1857 - end-to-end Story regression oracle verification.
• Require story-to-dev-unit completeness before claiming regeneration coverage #2358 - Story-to-dev-unit topology completeness.
• pdd bug RCA must separate underlying defect from detectability gap #1462 - separate underlying defects from detectability gaps.
• Generation quality umbrella: grounded requirements, validated interfaces, and outcome-based tests #1466 - generation-quality umbrella.
• Docs/Tooling: Add explicit waiver model for unchecked contract obligations #832 - explicit waivers for known unchecked obligations.
• feat: derive behavior contracts from source-truth edits before materialization #1470 - source-anchored behavior contracts.
• feat: emit clause-to-code traces and gate incomplete PDD candidates #1476 - clause-to-code trace and incomplete-candidate gates.
• pdd bug: generated tests must prove the user-reported outcome, not internal implementation details #1461 / Generate deterministic regression tests from a user story (pdd test --from-story) #1700 - downstream tests must prove user-observable outcomes.

[prompt-driven-github]

🚀 Job Queued!

Job ID: NX2AqPAU3CFvDRBFTjXF
Triggered by: @Serhan-Asad
Label: pdd-test

View Live Progress

Budget cap: none

You can add or change a cap by commenting:

/strength 0.7
/pdd budget N
/pdd settings
/pdd stop

Run controls require write access; /pdd settings is read-only.

[prompt-driven-github]

[Serhan-Asad]

/pdd stop

[prompt-driven-github]

PDD Process Terminated

No further automated comments will be posted. Re-add the trigger label to restart with your updated instructions.

[prompt-driven-github]

PDD Stopping

PDD is stopping this standalone run. If it was paused for a provider retry, the remaining retries are cancelled now; if it was actively running, the process will stop on its next poll.

Standalone runs have no saved state to resume. If this run was paused for a provider retry, you can re-add the the trigger label label now (optionally with a different pdd-* model label) to start fresh. If it was actively running, wait for it to stop before re-adding labels.