Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -296,7 +296,8 @@ Living tracker of projects shipped using this wizard. **Rule:** only list projec
| 235 | ~~Research: Thoughtworks "AI Evals" methodology vs our existing E2E pipeline~~ ✅ DONE 2026-05-04 — see [`.reviews/research-235-ai-evals.md`](.reviews/research-235-ai-evals.md) | **Verdict: NO-GO.** Audit confirms the article is methodology-only (no tools/frameworks named); every layer Thoughtworks describes (pre-deployment validation, post-deployment monitoring, quality gates, continuous oversight, performance consistency, output accuracy, error-mode catching, "dynamic nature of AI") already has a working analog in our pipeline (Tier 1/2 evaluator + score-history + CUSUM + token-spike-check + adversarial cross-model review + SDP scoring). The only candidate gap is bias/alignment evaluation, which is explicitly out of scope (the wizard is SDLC enforcement, not LLM ethics — that belongs to model providers and domain-specific tools). Pattern continues with #76 (Promptfoo NO-GO) and #77 (constrain-to-playbook NO-GO): external-methodology audits keep validating our pipeline is mature; revisit only when an external source points to a *specific* technique we don't have. **Original question:** does Thoughtworks' AI Evals framework ([thoughtworks.com/en-us/insights/decoder/a/ai-evals](https://www.thoughtworks.com/en-us/insights/decoder/a/ai-evals)) add anything we don't already have in our CI/CD? User call-out 2026-05-01: "should we be using AI Evals like Thoughtworks defines? if so add to roadmap or research." |
| 236 | ~~Audit: functional bloat hunt across wizard surface~~ ✅ **PHASE-PARTIAL DONE 2026-05-05; +tracked-artifact sweep 2026-05-06.** Three token-amplifier fixes shipped (v1.69.0 BASELINE-once-per-session ~12K saved/heavy-session, v1.70.0 TDD CHECK-once-per-session ~0.5-1.5K saved, v1.71.0 SDLC SKILL.md Cross-Model Review trim 4995 → 4568 tokens) plus one **orphan deletion** (`scripts/audit-subagent-models.sh`, 85 lines, 0 references, never in npm `files` list). **v1.73.0 (PR #331)** added category-(d)-adjacent sweep: -460 LOC across 15 stale `.reviews/` round-N + preflight artifacts plus `plans/CATCHUP.md` — all gitignore-shadowed leftovers from pre-`.gitignore` commits, none load-bearing. Live evidence in maintainer session: BASELINE block disappeared from system-reminders mid-conversation when sentinel landed. **Categories still not exercised:** (b) hooks (no nudge proven noise), (c) tests redundancy pass, (f) ROADMAP archive-below-fold. Re-trigger when a maintainer says "this repo is hard to find things in" again. **Original scope below for re-runs:** Recurring "does this still earn its keep?" pass. **Distinct from neighbors:** #220 token-bloat is size-per-file (chars/tokens); #19 Monthly GC is "weak tests / test fluff." This is **per-component existence justification**. **Method:** Prove-It style — for each candidate, attempt `git rm` on a branch, run full test suite. If nothing fails, that's a real bloat candidate. **Trigger:** quarterly. User call-out 2026-05-04: "bloat sucks." |
| 302 | User-level setup-wizard + repo-local lifecycle split — **DESIGN-CERTIFIED, implementation deferred** | GH issue #302 proposes splitting the wizard so `setup` lives user-level/global and `sdlc`/`update`/`feedback` stay repo-local ("install once, bootstrap anywhere"). Cross-model design review completed 2026-05-20 (Codex gpt-5.5 xhigh, [`.reviews/302-design-codex-review.md`](.reviews/302-design-codex-review.md)) scored Claude's first-pass analysis 5/10 NOT CERTIFIED and replaced it with a concrete channel contract. **Decided channel split:** plugin channel = user-level/global path (already exists: `.claude-plugin/plugin.json` + `hooks/hooks.json` with `${CLAUDE_PLUGIN_ROOT}`); npm/npx = explicit repo-local bootstrap/update path; no npm `postinstall` writing to `~/.claude/` (fragile under CI/containers/read-only HOME/Windows). **Design questions answered:** (1) `setup` global via plugin, not also copied repo-local (precedence #338 makes the global copy dead weight when both exist); preserve auditability via `SDLC.md` metadata or install manifest, not duplicate same-name skill. (2) `feedback` global via plugin, can still read repo-local `SDLC.md` at invocation time. (3) `update-wizard` stays repo-local (reads `SDLC.md` metadata, runs drift detection, mutates project files). (4) `npm install -g` installs the CLI binary only — no lifecycle scripts. (5) Migration: leave existing repos unchanged; new split is opt-in only; later major release can stop copying `setup`/`feedback` repo-local for new installs. **Hard constraint Claude missed:** "global setup auto-triggers anywhere" is FALSE — `hooks/instructions-loaded-check.sh:14-22` and `hooks/sdlc-prompt-check.sh:27-35` exit silently when `SDLC.md`/`TESTING.md` don't exist, and `tests/test-hooks.sh:2507/2522` actively assert that silence. A plugin-only fresh-repo bootstrap signal must be designed before the global skill can satisfy the "always available" promise from the issue. **Existing dual-channel surface to respect:** `cli/init.js:286-302` blocks npm-on-plugin (duplicate `/update-wizard`), `hooks/instructions-loaded-check.sh:195-206` warns on dual install. Adding npm-global skills would create a third overlapping surface — deprecate one channel first. **Implementation sequence (5 steps, do NOT start without dedicated session):** (1) lock the channel contract in `CLAUDE_CODE_SDLC_WIZARD.md` + ARCHITECTURE.md; (2) doc the fresh-repo trigger limitation + migration story in #302 comment; (3) prototype plugin-only fresh-repo bootstrap signal + assert current silent-hook tests still pass; (4) write global-skill collision/backup/update tests if any installer touches `~/.claude/skills/`; (5) only then alter `init` behavior for new installs, leaving existing ones intact. **Why deferred 2026-05-20:** session shipped v1.74.0 → v1.75.0 → v1.75.1 (Trusted Publishing migration), maintainer fatigued, #302 is architectural — needs a fresh planning session. **Trigger to revisit:** explicit user request OR a second repo install where "install once" friction is felt acutely. |
| 347 | Goal-mode checkpoint workflow (Codex `$gdlc` equivalent) — **RESEARCH-DONE, implementation deferred** | GH issue #347 asks whether Claude Code has a native primitive for long-running goal-bound work (Codex `$sdlc + $gdlc` pattern with persistent constraints, checkpoint cadence, explicit stop boundaries). Research completed 2026-05-23 ([`.reviews/347-goal-mode-research.md`](.reviews/347-goal-mode-research.md)) via claude-code-guide subagent against CC docs current as of 2026-05-23. **Verdict: NO native primitive.** CC has no first-class abstraction for a long-running multi-session objective with enforced lifecycle constraints. Sessions are independent JSONL files; persistence is indirect via `CLAUDE.md`, auto memory, hooks. **Per-feature mapping:** TaskCreate = per-session only; subagents = per-invocation context only; skills = standing instruction for current session only; hooks = YES the deterministic enforcement layer; plan mode = per-turn only; `CLAUDE.md` = context not enforced configuration; no `/goal` command on roadmap. **Closest existing primitive:** user-invocable skill + `SessionStart` hook + project `GOAL.md` — exactly what the wizard already does for SDLC enforcement, just unwrapped for goal-mode semantics. **Prove-It Gate verdict:** absorb into existing `/sdlc` skill as a "Long-Running Goal Mode" section + `GOAL.md`/`HANDOFF.md` templates rather than building a new `/goal-start`/`/goal-handoff`/`/goal-stop` skill trio. Default to absorption per #236 functional bloat principle; promote to dedicated skill only if absorbed form proves too easy to ignore. **Implementation sequence (5 steps, do NOT start without dedicated session):** (1) add `GOAL.md` + `HANDOFF.md` templates to `cli/templates/` and prompt for them in setup-wizard's long-running-project branch; (2) extend `/sdlc` skill with "Long-Running Goal Mode" section (objective binding, checkpoint cadence, stop-boundary enforcement); (3) extend `hooks/instructions-loaded-check.sh` to inject `GOAL.md` contents via `additionalContext` when present; (4) extend `hooks/sdlc-prompt-check.sh` to re-assert stop boundary if `GOAL.md` declares one and context is >75% full; (5) quality tests proving goal injection fires only when `GOAL.md` present, stop-boundary reassertion fires only past threshold, `HANDOFF.md` write is idempotent across resume sessions. **Why deferred 2026-05-23:** only one demand signal (issue author); Prove-It Gate "Default answer is NO unless proven YES" applies. Per #233 pattern, build when a second user files for it OR when the maintainer's own work hits the goal-mode gap. **Trigger to revisit:** second user request, OR maintainer files a session-spanning project where the absence of `GOAL.md` is visibly painful. |
| 347 | Goal-mode checkpoint workflow (Codex `$gdlc` equivalent) — **CORRECTED 2026-05-24: native `/goal` exists, scope shrunk to ~30-line skill wrapper** | GH issue #347 asks whether Claude Code has a native primitive for long-running goal-bound work (Codex `$sdlc + $gdlc` pattern with persistent constraints, checkpoint cadence, explicit stop boundaries). **2026-05-23 research was WRONG** — claude-code-guide subagent said no native primitive, but CC **v2.1.139 shipped native `/goal`** (confirmed via raw changelog curl; docs at [code.claude.com/docs/en/goal.md](https://code.claude.com/docs/en/goal.md); follow-up fixes v2.1.140 hook-disabled hang + v2.1.143 subagent race). Corrected research at [`.reviews/347-goal-mode-research-CORRECTED.md`](.reviews/347-goal-mode-research-CORRECTED.md). **What `/goal` does:** session-scoped, evaluator-driven (Haiku default judges transcript after each turn yes/no), survives `--resume` but not `/clear`, no disk writes, no native turn/time cap. UX = `/goal <condition>` + `/goal` (status) + `/goal clear`. **Old verdict OBE.** **Corrected scope = ~30-line skill wrapper in existing `/sdlc` (NOT a `GOAL.md` template — `/goal` writes nothing to disk).** Per [`.reviews/347-goal-mode-research-CORRECTED.md`](.reviews/347-goal-mode-research-CORRECTED.md), the wizard should add: (a) **pre-flight checklist** — workspace trusted, hooks not disabled at any settings layer, CC ≥ v2.1.143; (b) **condition-writing guidance** mirroring SDLC reporting standards (measurable end state + check + constraints + hard turn/time bound since `/goal` has no native cap); (c) **compose-with-hooks note** — `UserPromptSubmit`/`SessionStart`/`PreCompact` fire normally inside the goal loop, so `sdlc-prompt-check.sh` + `precompact-seam-check.sh` keep gating each turn; (d) **resume caveat** — `--resume` resets turn/time counters; (e) **anti-pattern callout** — don't use `/goal` for "doneness" the evaluator can't see in the transcript (evaluator can't call tools). **Status:** Actionable now (qualifies as "low-cost cleanup" under demand-signal-first gate — ~30 lines, zero-API, builds on existing skill, corrects a wrong public answer). **Original 5-step plan + `GOAL.md` template scaffolding OBE.** **Meta-lesson captured in CORRECTED doc:** require explicit citation of an authoritative source (docs index, raw changelog grep) before accepting a "feature does not exist" claim — negative claims are easier to fake than positive ones. **Process gap also caught:** auto-update PR workflow stopped flagging features at CC v2.1.118 (2026-04-23, last PR #210) because #231 Phase 3d (v1.54.0) gutted the in-CI LLM-ranker to $0. Manual replacement (`claude --print --allowedTools "WebFetch,Read,Bash" "$(cat .github/prompts/analyze-release.md)"`) was supposed to run weekly on Max — it didn't, which is how `/goal` slipped past unnoticed for ~5 weeks. See #350 for the cadence fix. |
| 350 | CC feature-discovery cadence — replacement-for-#231 ranker has no enforced cadence | **Process gap discovered 2026-05-24 via #347 correction.** #231 Phase 3d (v1.54.0, 2026-04-29) deleted the in-CI Claude ranker from `weekly-update.yml` to take it to $0/week. The fallback was "maintainer runs `claude --print "$(cat .github/prompts/analyze-release.md)"` weekly on Max." That ran zero times in 5 weeks. Consequence: native CC `/goal` shipped in v2.1.139 (~mid-May) and was missed until a user asked a related question 2026-05-24 — wizard then published wrong research saying "no /goal exists." 39 CC versions accumulated since the last auto-PR (v2.1.118 → v2.1.150). **Scope (Actionable now, low-cost cleanup):** (a) restore a thin cron — `weekly-update.yml` still runs detection ($0 GH API call); add a single follow-up step that opens a "CC version drift" GitHub issue when our `<!-- SDLC Wizard Version -->` baseline is >5 minor versions behind latest npm — pure GitHub API, no LLM, no API spend; (b) the issue body links the maintainer to the on-Max command + the changelog URL + a one-line "what to do" reminder; (c) session-start `instructions-loaded-check.sh` already has the staleness nudge (#196 — "N minor versions behind") — verify it actually fires when the gap is ≥3 minor AND link to the analyze-release runbook in the loud branch. **Why this beats #85 Phase 2** (which we killed 2026-05-24): #85 Phase 2 was "auto-rank features in CI via LLM" — expensive and conflicted with #231. This is "open a stale-baseline issue when we detect drift" — cheap, GH API only, just a maintainer reminder. **Status:** Actionable now. **Trigger to do:** immediately, paired with #347 since both came from the same discovery. |

## Review Pipeline

Expand Down