Skip to content

Automated test documents contain the test, and outcomes go to the Knowledgebase #401

Description

@DutchJaFO

Background

Developer rules (2026-09-17): an automated test document contains the test and nothing else, and an observation is a test result — never part of the test plan, except where it states an expected result. Outcomes feed the Knowledgebase, which is how an operator finds the cause of what they are seeing; they do not belong in the document that specifies the test.

Two shapes breach that across the suite:

  • 55 documents carry an ## Observed effect section. The suite index's Test outcomes feed the Knowledgebase section is what instructed them to, and its intent is right — the destination was wrong.
  • Documents carry prose that is not the test: rewrite histories ("Rewritten 2026-09-08, because the previous query could not pass"), figures measured in past runs, justifications for approaches since abandoned, and reasoning about defects already fixed. import-and-staged-actions/14 alone holds several pages of it.

docs/knowledgebase/ now exists, with a template and three entries, so relocated outcomes have a destination (#397).

What needs to be done

  1. Remove every ## Observed effect section, writing anything of operator value into a docs/knowledgebase/ entry first.
  2. Remove prose that is not needed to run or judge the test — history, past measurements, superseded rationale. What stays: the header fields, Preconditions, Determinism as a list of pinned variables, the steps with their expected results, On failure guidance, and Cleanup.
  3. Reword the index's Test outcomes feed the Knowledgebase section to say where an outcome goes: a Knowledgebase entry, never the document.
  4. Guard it, so the shape cannot come back unnoticed.

Expected tests

Test class / document Test method / what it verifies Starts
RepositoryStructureTests NoAutomatedTestingDocumentRecordsItsOwnOutcome ❌
RepositoryStructureTests TheSuiteIndexSendsOutcomesToTheKnowledgebase ❌

Notes

  • A document's expected results stay exactly as they are: naming the exception type a step must produce is an expectation, not an observation.
  • Removing a measured figure that a step currently asserts against is out of scope here — where a figure is load-bearing, it is either an expectation the step keeps or a defect for its own issue.

Definition of done

  • All expected tests listed above start red before implementation
  • All requirements implemented
  • All expected tests pass (green)
  • No regression in related tests
  • Findings summarised in a closing comment

Activity

  1. added this to the Notification system milestone on Sep 17, 2026
  2. added 2 commits that reference this issue on Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions