Skip to content

archive --yes proceeds on self-reported task status — should archive be able to require evidence? #1652

Description

@dahankzter

What happens

openspec archive --yes archives a change even when no task in tasks.md is
checked off, and even when nothing in the repo works. Reproduction on 1.9.0:

$ openspec new change demo
$ # ...write proposal.md + specs/thing/spec.md...
$ cat openspec/changes/demo/tasks.md
## 1. Work

- [ ] 1.1 do the thing

$ openspec archive demo --yes
Warning: 1 incomplete task(s) found. Continuing due to --yes flag.
Specs updated successfully.
Change 'demo' archived as '2026-08-14-demo'.

The spec deltas are merged into openspec/specs/ and the change is filed away.
No code was written.

Why I think this is worth a design discussion rather than just a bug

The check that exists (archive.ts, the incompleteTasks block) is doing what
it was designed to do, and interactively it does prompt. The gap is narrower and
more interesting than "the warning is too weak":

tasks.md checkboxes are written by whoever did the work. When that is an
assistant, the boxes record an intention to have finished, not evidence of it.
A box gets ticked because a model believes the step is done. So the strongest
signal archive currently has is a self-report, and --yes — which is the normal
flag in any script or CI job — accepts it unconditionally.

That matters more for OpenSpec than for most tools, because archiving is not a
bookkeeping step: it merges deltas into the main specs. openspec/specs/ is
meant to describe how the system actually behaves, and after a --yes archive
over unchecked boxes it describes how the system was intended to behave. The
two diverge silently, and the spec is the artifact everything downstream trusts.

Validation doesn't cover this either, by design — openspec validate --strict
proves the documents are well-formed, never that they are true of the code. Both
of the existing gates are therefore about documents and intentions; neither can
observe the repo.

The question

Would you accept archive being able to require evidence — a command that must
exit 0 — before it files a change away?

The smallest version I can see fits the config shape that already exists.
operations.archive currently holds guidance, which is advice handed to an
assistant. This would add a sibling that is executed:

operations:
  archive:
    guidance:                    # existing: advice for the assistant
      - Summarize the archive outcome
    checks:                      # new: commands that must exit 0
      - npm test
      - npm run lint

Absent config, nothing changes. --yes would not bypass checks — it answers
prompts, and a failing command isn't a prompt — with --skip-checks as the
deliberate override.

The objection I'd raise against my own proposal

This would be the first time OpenSpec executes commands from a repo file.
Clone a hostile repo, run openspec archive, and its config.yaml runs
arbitrary code. That is the same trust model as npm lifecycle scripts or git
hooks, and it only fires on an explicit user-initiated archive — but OpenSpec
doesn't execute anything today, so this genuinely widens the surface, and that's
a fair reason to say no or to want a guard (first-run confirmation, an opt-in
global setting, an env var).

I have a working implementation with tests if there's appetite, but I'd rather
hear whether you want this in OpenSpec at all, and how you'd want the execution
guarded, before opening a PR. Happy to drop it if the answer is that archive
should stay document-only.

Activity

  1. SoLoHiC commented on Sep 10, 2026

    @SoLoHiC

    Related work in #1791 separates task completion from archive readiness, and #1732 addresses skipped verification being reported as passing. This proposal raises a further question: what evidence should a project's archive policy accept?
    It may help to separate two decisions:

    • Execution: does OpenSpec run configured commands, or does the project's CI/workflow run them?
    • Acceptance: which results are required, what change/revision do they cover, and are they still applicable when archive runs?

    For example, a passing test run followed by an implementation change should not silently satisfy the same archive check. Conversely, a documentation-only follow-up need not invalidate every result if the project has an explicit reuse policy.
    An externally produced receipt is one possible option if command execution is undesirable in core. It would still need a trust policy and binding to the relevant inputs; merely finding a receipt file would recreate the checkbox problem.
    Disclosure: I maintain RVTF, whose OpenSpec companion adapter separates progress, evidence and closure. Its evaluated contract is pinned to OpenSpec 1.9.0; I'm sharing the design distinction here, not claiming current-version enforcement.

  2. MS8AT commented on Sep 21, 2026

    @MS8AT

    Building on the distinction above between execution and acceptance, I would like to contribute a small documentation-only recipe to docs/reviewing-changes.md.

    It would show how to map a requirement/scenario to evidence for the relevant source or artifact and environment, with a fictional receive-traffic timeout example. Negative review cases include an outdated requirement revision, a different binary, a wrong environment, duplicate evidence, and an unknown result.

    This would not introduce command execution, a mandatory evidence schema, or archive enforcement. The recipe would explicitly distinguish externally verifiable evidence from an assistant-authored PASS, and leave enforcement to the project's existing CI/release policy. It would also cover when evidence reuse is reasonable for unrelated documentation changes.

    The contribution contains only general guidance and synthetic examples, with no private project configuration, endpoints, device identifiers, or logs. I will link a docs-only PR here. This complements the existing discussion; it does not resolve the core design decision in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    design-reviewNeeds product/design decision

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions