Skip to content

discovery(tests): orchestration suite has 5 pre-existing failures — checkpoint test rot (_checkpoint_manager rename) + workflow_memory isolation pollution #11754

Closed

Description

@mrveiss

Discovered during

Implementation of #6828 (AgentRegistry consolidation, 2026-07-15) — surfaced when running the full autobot-backend/orchestration/ suite as a pre-commit verification gate. Reproduced identically on a clean Dev_new_gui tip (7a95b99), so both are pre-existing and unrelated to the #6828 diff.

Finding 1 — workflow_executor_checkpoint_test.py: 3 deterministic failures (test rot)

autobot-backend/orchestration/workflow_executor_checkpoint_test.py:107:
    executor._checkpoint_manager.save = save_spy
E   AttributeError: 'WorkflowExecutor' object has no attribute '_checkpoint_manager'. Did you mean: '_checkpoint_resumer'?

Failing tests (always, even in isolation):

  • test_checkpoint_failure_does_not_propagate_in_execute_step_with_agent
  • test_checkpoint_success_called_once_on_happy_path
  • test_checkpoint_not_called_when_step_fails

The production attribute was renamed _checkpoint_manager → _checkpoint_resumer without updating the test's spy target. The tests need re-pointing at _checkpoint_resumer (and re-verifying they still assert the intended checkpoint semantics rather than just the attribute name).

Finding 2 — workflow_memory_test.py: 2 ordering-dependent failures (test isolation)

  • TestWorkflowMemoryAutoInjection::test_prior_findings_injected_into_context
  • TestWorkflowMemoryAutoInjection::test_empty_memory_not_injected

Pass when the file runs alone (34 passed), fail when run as part of pytest autobot-backend/tests/orchestration/ autobot-backend/orchestration/ ... — cross-test state pollution (likely a shared singleton/module-level cache mutated by an earlier test). Needs a fixture-level reset of whatever seam the auto-injection reads.

Evidence

# worktree AND clean base tip both:
python3 -m pytest autobot-backend/orchestration/workflow_executor_checkpoint_test.py -q
# -> 3 failed
python3 -m pytest autobot-backend/orchestration/workflow_memory_test.py -q
# -> 34 passed (alone)
python3 -m pytest autobot-backend/tests/orchestration/ autobot-backend/orchestration/ autobot-backend/services/mesh_brain/ -q
# -> 5 failed, 708 passed

Why it matters

Deterministic red tests in the orchestration suite mask real regressions in checkpointing, and ordering-dependent failures make full-suite runs untrustworthy as a merge gate.

Activity

  1. mrveiss commented on Jul 16, 2026

    @mrveiss
    OwnerAuthor

    Closed — merged to Dev_new_gui. Commit: 659781f (PR #11788). Checkpoint spies re-pointed to the #6827 _checkpoint_resumer delegation seam (3 tests); circuit-breaker state bleed fixed via new orchestration conftest autouse reset_all_circuit_breakers() (established seam, no reordering/flaky marks). Evidence: full orchestration group 713 passed / 0 failed (was 5 failed / 708); independent re-run 63 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions