Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions .bootstrap/config/core.toml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ project_root = ".."
[kits.sdlc]
format = "CFS"
path = "config/kits/sdlc"
version = "v1.0.0"
version = "1.0.0"
source = "github:constructorfabric/studio-kit-sdlc"

[kits.sdlc.resources]
Expand Down Expand Up @@ -106,19 +106,17 @@ path = "config/kits/sdlc/workflows/migrate-openspec.md"

[kits.sdlc.source_provenance]
source_type = "github"
resolver_mode = "latest_release"
resolution_basis = "github_release"
resolver_mode = "default_branch"
resolution_basis = "github_default_branch"
requested_ref = "latest"
resolved_ref = "v1.0.0"
canonical_source = "github:constructorfabric/studio-kit-sdlc"
effective_source = "github:constructorfabric/studio-kit-sdlc"
verified = "verified"
freshness = "fresh"

[kits.sdlc.content_identity]
commit_sha = "78b02ac"
resolved_ref = "v1.0.0"
identity = "constructorfabric/studio-kit-sdlc@v1.0.0#78b02ac"
identity = "constructorfabric/studio-kit-sdlc@#78b02ac"

[kits.sdlc.local_metadata]
conf_version = "1.0.0"
Expand Down
27 changes: 27 additions & 0 deletions .cspell.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"version": "0.2",
"words": [
"PDSL",
"Rcons",
"Rpb",
"TOCTOU",
"subagent",
"subagents",
"plaintext",
"worktree",
"worktrees",
"cfs",
"toml",
"yaml",
"frontmatter",
"semver",
"kebab",
"brownfield",
"greenfield"
],
"ignorePaths": [
".bootstrap/**",
"node_modules/**",
".prs/**"
]
}
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ Spaider.code-workspace
.bootstrap/.plans/analyze-path-current-branch-vs-main-comprehensive/
.bootstrap/.plans/implement-thin-orchestrator-workflows/
.bootstrap/.plans/implement-shared-context-pack-pdsl-migration/
.bootstrap/.plans/audit-studio-prompts/

# Artifacts
.prs/*
Expand Down
6 changes: 6 additions & 0 deletions .markdownlint.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"MD013": {
"code_blocks": false,
"tables": false
}
}
58 changes: 53 additions & 5 deletions requirements/prompt-bug-finding.md

Large diffs are not rendered by default.

49 changes: 41 additions & 8 deletions requirements/prompt-engineering.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
cf: true
type: requirement
name: Prompt Engineering Review Methodology
version: 1.4
purpose: Systematic methodology for reviewing and improving agent instructions with compact-prompts optimization, interaction UX quality, and router-based decomposition
version: 1.5
purpose: Systematic methodology for reviewing and improving agent instructions with evidence-backed constraint framing, compact-prompts optimization, interaction UX quality, and router-based decomposition
---

# Prompt Engineering Review Methodology
Expand Down Expand Up @@ -45,9 +45,11 @@ purpose: Systematic methodology for reviewing and improving agent instructions w

Agent instructions are executable policy for agent behavior and user interaction. Review them like software: classify the artifact, test for ambiguity, verify structure, identify missing contracts, detect anti-patterns, manage context budget, confirm testability, check interaction UX, check model ergonomics, then synthesize prioritized fixes.

**High-priority rule**: for analysis and generation work, aggressively reduce loaded context whenever behavior, determinism, constraints, safety, output contracts, and recovery rules remain intact.

**CRITICAL interaction UX rule**: whenever instructions ask the user for input, confirmation, or a choice, review whether the prompt explains why the input is needed, what the user is expected to provide, what each option leads to, which option is suggested in the current context, and exactly how the user should reply.
**Review stance**:
- Reduce loaded context whenever behavior, determinism, constraints, safety, output contracts, and recovery rules stay intact.
- Prefer positive, action-oriented requirements. If blocked behavior matters, pair the prohibition with the required alternative and a verification check.
- Keep one response surface to `3-7` primary rules when possible; allow up to `10` only when rules are short, independent, and checked. Beyond that, decompose or validate.
- For every user ask, confirm the prompt explains why the input is needed, what the user should provide, what each option does next, which path is suggested, and how to reply.

## Layer Map

Expand Down Expand Up @@ -88,6 +90,10 @@ Agent instructions are executable policy for agent behavior and user interaction

**Framing**: prefer positive requirements; if a negative is necessary, pair it with the required alternative; distinguish `MUST NOT` / `NEVER` from `SHOULD NOT` / `AVOID`.

**Negative-constraint handling**: flag negative-only instructions (`do not X`, `never Y`, `avoid Z`) unless they also state the desired replacement behavior. Name exact blocked tokens only when verification depends on exact matching; otherwise prefer category-level constraints plus the required replacement behavior.

**Instruction count**: count active requirements the model must satisfy in one response. Treat `> 7` concurrent requirements as a risk and `> 10` as a decomposition trigger unless the prompt includes a validator, checklist, or iterative self-refinement loop.

**Priority**: critical rules are marked (`MUST`, `REQUIRED`, `CRITICAL`), optional rules are marked (`MAY`, `OPTIONAL`, `CONSIDER`), and importance hierarchy is obvious.

**Compact clarity rules**: use short imperative sentences; front-load trigger + action + object (`WHEN X, do Y to Z`); use explicit nouns and verbs; replace vague wording with measurable limits or decision rules; keep stable terminology; remove filler and repeated restatements; prefer bullets, tables, and checklists over narrative; keep only examples that change behavior or clarify edge cases.
Expand Down Expand Up @@ -133,6 +139,9 @@ Agent instructions are executable policy for agent behavior and user interaction
| `AP-LONG-WINDED` | The same rule is padded with prose, repetition, or bloated examples. |
| `AP-CONFLICTING` | Requirements contradict one another. |
| `AP-IMPOSSIBLE` | Not all requirements can be satisfied simultaneously. |
| `AP-NEGATIVE-ONLY` | A prohibition says what not to do without stating the required alternative behavior. |
| `AP-FORBIDDEN-PRIMING` | A prompt repeatedly names exact blocked tokens or labels when category-level wording would preserve the rule. |
| `AP-INSTRUCTION-DENSITY` | More than `7` active requirements compete in one response surface, or more than `10` are present without decomposition or validation. |
| `AP-NO-ROUTER` | Multi-step or branching instructions lack a compact router/index that says what may load next and when. |
| `AP-OVERSIZED-RESOURCE` | A loadable instruction resource, module, or deliberate slice exceeds `200` lines. |
| `AP-MONOLITHIC-STEP` | Multiple steps, branches, or modes are bundled into one loadable unit instead of decomposed into routeable modules. |
Expand All @@ -142,10 +151,11 @@ Agent instructions are executable policy for agent behavior and user interaction
| Code | Detect when |
|---|---|
| `AP-CONTEXT-BLOAT` | Excessive context dilutes priorities. |
| `AP-SYSTEM-PROMPT-BLOAT` | A system prompt violates `6.1.3`: always-on text is `> 200` lines or embeds conditional blocks that should be modular. |
| `AP-SYSTEM-PROMPT-BLOAT` | A system prompt violates the compact-prompt always-on budget rule: always-on text is `> 200` lines or embeds conditional blocks that should be modular. |
| `AP-CONTEXT-STARVATION` | Critical context is missing. |
| `AP-CONTEXT-DRIFT` | Required context may be lost through compaction or long sessions. |
| `AP-BURIED-PRIORITY` | Critical rules are hidden instead of surfaced early and scannably. |
| `AP-LOST-MIDDLE` | Critical instructions or references sit in the middle of a long prompt where attention may degrade. |
| `AP-VAGUE-REFERENCE` | References such as `the above` or `this` have no clear antecedent. |
| `AP-ASSUMES-MEMORY` | The document assumes the agent will remember earlier turns. |
| `AP-NO-CHECKPOINT` | Long workflows lack state checkpoints. |
Expand Down Expand Up @@ -261,7 +271,9 @@ A **loadable instruction resource** is any file, module, or deliberate contiguou

**Lifecycle**: specify what loads at start, what loads on demand, what can be summarized when context is low, what must never be dropped, how critical state survives compaction, what belongs in files vs working memory, and how context loss is detected and recovered.

**Attention management**: repeat or reinforce critical instructions, visually emphasize important sections, keep guardrails in a dedicated section, avoid too many competing instructions, group related rules, and separate low-priority content.
**Attention management**: place critical instructions near the beginning or end of the active prompt surface, repeat or reinforce only the highest-risk constraints, visually emphasize important sections, keep guardrails in a dedicated section, avoid too many competing instructions, group related rules, and separate low-priority content.

**Instruction-density audit**: list every active instruction that must be followed in the next response. Merge duplicates, remove non-operative prose, and convert large rule sets into a router plus current-phase checklist. If the active set remains `> 10`, mark `AP-INSTRUCTION-DENSITY` and recommend decomposition.

## L7: Testability Assessment

Expand All @@ -271,6 +283,8 @@ A **loadable instruction resource** is any file, module, or deliberate contiguou

**Built-in checks**: include validation criteria, a pre-completion self-check, checklist formatting for critical steps, and proof-of-work requirements when appropriate.

**Constraint self-check**: when a prompt has `5+` active constraints, require the agent to verify the final answer against the constraints before completion. The self-check may be internal or visible, but the final output contract must make compliance observable when failure risk is high.

**Interactive UX checks**: when the document asks the user to choose or confirm, verify that tests can confirm all of the following from the emitted prompt alone: why the input is needed, what reply format is accepted, what each option does, and whether any suggested option is anchored to the current context.

**When a workflow requires terminal prompts or final handoff blocks, the pre-completion self-check should verify that those exact blocks were emitted before the response may end.**
Expand Down Expand Up @@ -313,6 +327,10 @@ A **loadable instruction resource** is any file, module, or deliberate contiguou

**Training alignment**: use familiar prompt patterns, an appropriate role/persona, and a style consistent with effective prompting.

**Positive action bias**: rewrite vague or negative rules into direct actions. Prefer `Do X when Y` over `Do not forget X`; prefer `If data is missing, say UNKNOWN` over `Do not hallucinate`.

**Constraint realism**: treat exact word counts, exact token counts, exact character limits, and long simultaneous rule sets as high-risk unless automated validation or post-processing is available.

**Graceful degradation**: define what happens on partial failure, whether the agent can recover without intervention, and when it must ask for help.

**Hallucination prevention**: require verification or citation, permit uncertainty, mark speculation, and use external tools for factual queries.
Expand All @@ -321,6 +339,8 @@ A **loadable instruction resource** is any file, module, or deliberate contiguou

**Conversation compatibility**: support multi-turn use, clarification requests, and mid-task scope changes.

**Examples over rule volume**: when format, tone, or structure matters, prefer one compact positive example over several overlapping prose rules. Keep examples small and remove any example that no longer changes behavior.

## L10: Improvement Synthesis

**Severity**:
Expand Down Expand Up @@ -393,10 +413,20 @@ When the deterministic gate is `SKIPPED`, do not describe semantic review, check

## References

This document is the authoritative working method. External sources informed its design, but the prompt surface here stays intentionally compact.
This document is the authoritative working method. External sources inform its design, but the prompt surface here stays intentionally compact.

**Companion methodology**: `prompt-bug-finding.md` for bug hunting, hidden failure modes, unsafe behavior, regressions, instruction conflicts, or root-cause analysis in prompts and agent instructions.

**Research and practice references**:

- OpenAI prompt engineering best practices: put instructions first, separate instruction from context, use specific desired formats, show examples, reduce imprecise wording, and state what to do instead of only what not to do. https://help.openai.com/en/articles/6654000-using-advanced-prompt-engineering-techniques
- `Semantic Gravity Wells: Why Negative Constraints Backfire`: negative constraints can fail because naming forbidden terms primes the model toward them. https://arxiv.org/abs/2601.08070
- `Curse of Instructions`: instruction following degrades as the number of simultaneous instructions increases; prompts with up to ten verifiable instructions show sharp all-instruction success loss, partially improved by self-refinement. https://openreview.net/forum?id=R6q67CDBCH
- `InFoBench`: complex instructions are better evaluated by decomposing them into simpler criteria and checking requirement-level compliance. https://arxiv.org/abs/2401.03601
- `FollowBench`: incrementally adding constraints exposes weaknesses in fine-grained constraint following. https://arxiv.org/abs/2310.20410
- `Lost in the Middle`: long-context models may underuse information placed in the middle of long inputs; critical rules should not be buried there. https://arxiv.org/abs/2307.03172
- `What Prompts Don't Say`: simply adding more requirements does not reliably improve performance because instruction-following capacity is limited and requirements may conflict. https://arxiv.org/abs/2505.13360

## Validation

Review is complete when:
Expand All @@ -408,6 +438,9 @@ Review is complete when:
- [ ] Implementation guidance provided
- [ ] Safe compact-prompts opportunities identified and prioritized for prompt/instruction documents
- [ ] Compact-prompts findings reported explicitly in the review output
- [ ] Negative-only instructions checked and paired with required alternative behavior where needed
- [ ] Active instruction count checked; `> 7` constraints treated as risk and `> 10` constraints decomposed or validator-backed
- [ ] Critical instructions checked for beginning/end placement rather than being buried in long-context middle sections
- [ ] For every user-facing interaction point, question purpose, option clarity, option outcomes, suggested-path quality, reply format, and fallback clarity were checked explicitly
- [ ] Required completion gates, terminal blocks, and false-completion paths were checked explicitly when the document defines a final response contract
- [ ] Verification plan included
2 changes: 1 addition & 1 deletion requirements/raw-input-overflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ purpose: Shared overflow routing rule for analyze and generate workflows

# Raw-Input Overflow Rule

If the direct user prompt plus all provided files exceeds `500` total lines, the agent MUST NOT silently continue in direct workflow mode. It MUST present an explicit choice between `(a)` switching to `/cf-plan` or `(b)` continuing in the current direct workflow with reduced guarantees. If the user chooses `/cf-plan`, preserve the same request scope and require the planner to materialize that raw input under `{cf-studio-path}/.plans/{task-slug}/input/` before decomposition. The planner MUST obtain explicit user approval before creating that directory or executing the write-capable `chunk-input` command shown below, and MUST pass `--include-stdin` when direct prompt text must be packaged together with provided files. If the user declines planning, the agent MAY continue in direct workflow mode only after explicitly warning that context overflow may reduce rule coverage, checklist coverage, or output quality. The explicit offer takes precedence over any later **single-context bypass check** (the Phase 1.2 estimate in `plan.md` that allows skipping plan compilation when the compiled estimate is `<= 500` lines): once the user accepts `/cf-plan` here, the planner MUST stay on the plan path even if that later check would otherwise allow a bypass.
If the direct user prompt plus all provided files exceeds `500` total lines, the agent MUST NOT silently continue in direct workflow mode. It MUST present an explicit choice between `(a)` switching to `/cf-plan` or `(b)` stopping before direct execution. If the user chooses `/cf-plan`, preserve the same request scope and require the planner to materialize that raw input under `{cf-studio-path}/.plans/{task-slug}/input/` before decomposition. The planner MUST obtain explicit user approval before creating that directory or executing the write-capable `chunk-input` command shown below, and MUST pass `--include-stdin` when direct prompt text must be packaged together with provided files. Direct-workflow continuation is allowed only when a downstream workflow explicitly preserves that branch after this choice; otherwise the fallback is `/cf-plan` or stop. The explicit offer takes precedence over any later **single-context bypass check** (the Phase 1.2 estimate in `plan.md` that allows skipping plan compilation when the compiled estimate is `<= 500` lines): once the user accepts `/cf-plan` here, the planner MUST stay on the plan path even if that later check would otherwise allow a bypass.

Canonical write-capable invocation (executed only after explicit approval):

Expand Down
4 changes: 2 additions & 2 deletions requirements/storytelling-phases.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Extract from the prompt:

### Step 2: Resolve mode (always-ask)

Methodology emits the 6-mode prompt (template in `{cf-studio-path}/.core/requirements/storytelling-modes.md`); waits for explicit user confirmation (Enter accepts the suggestion, or pick by number/name). Mode resolution is interactive every session — intent verbs / KIND defaults / `default_mode` from preferences.json only inform the suggested default, never bypass the prompt.
Methodology emits the 6-mode prompt (template in `{cf-studio-path}/.core/requirements/storytelling-modes.md`); waits for explicit user confirmation. Reply contract for both the mode prompt and the disposition prompt: pick by number, exact name, or Enter for the suggestion. Mode resolution is interactive every session — intent verbs / KIND defaults / `default_mode` from preferences.json only inform the suggested default, never bypass the prompt.

After mode resolution but BEFORE role/audience derivation, methodology emits the **artifact disposition** prompt (template + resolution rules in `{cf-studio-path}/.core/requirements/storytelling-preferences.md` Artifact Disposition). The list of accumulating artifact types in the prompt depends on the resolved mode (review adds review-comments; all modes have open-questions and bookmarks). The methodology MUST wait for explicit user confirmation; the project `artifact_disposition` preference informs the suggested default but does NOT bypass the prompt.

Expand Down Expand Up @@ -347,7 +347,7 @@ Hard rules, enforced inside every portion:

1. **Information SHALL come from the input only**: target artifact / codebase region + its registered linked artifacts (parents/children fetched in E0) + the user's prompt. **Nothing else.**

2. **No invention. No agent-initiated gap markers.** If a claim cannot be grounded, the rule is **silently skip**. MUST NOT insert `[?]` markers in the methodology's narrative; MUST NOT push to open-questions buffer on its own initiative. Open-questions entries created **only** when the user asks a question the input cannot answer (see User input handling). NEVER paraphrase domain knowledge as if from the input.
2. **Ground or omit.** If a claim cannot be grounded, omit it rather than fabricating. MUST NOT insert placeholder gap markers in the methodology's narrative or push to the open-questions buffer on its own initiative. When the user directly asks for unavailable information, say the input does not cover it and create an open-question entry per User input handling. Open-questions entries are otherwise created **only** when the user asks a question the input cannot answer. NEVER paraphrase domain knowledge as if from the input.

3. **Source reference required** for every non-trivial claim, as a **clickable Markdown link** (plain-text refs like `(DESIGN.md §4.2)` are forbidden):
- Unregistered file with heading: `(see [{file} §{section}]({path}#{anchor}))` — ex: `(see [DESIGN.md §4.2 Data Model](DESIGN.md#42-data-model))`
Expand Down
Loading
Loading