Repository navigation
Test suite accumulates workflow-run files in the repo's .nodus/, and LocalWorkflowStore rescans all of them — the suite destabilizes its own later runs #380
Description
Activity
- added a commit that references this issue
on Aug 15, 2026 Part 1 landed in #381 (
72fc8f8) — the accumulation is stopped. Part 2 (bounding the store's cost) stays open here.Verified by counting rather than by reasoning:
repo runs/afterfull pytest tests/— before41 files full pytest tests/— after0 nodus_gate --all— beforefiles added nodus_gate --all— after0 The first attempt was wrong and is worth recording. Pointing the default store at a temp root for the whole session — the obvious reading of "use a temp store" — broke 26 tests. Several build a project directory,
chdirinto it, and assert the default runner wrote under that root. CWD-relative behaviour is the documented contract and those tests are right to check it; the defect was never that files get written, only that they outlive the session. So the fix is cleanup, not redirection: a session fixture removes what the run added and leaves anything that was already there.The doc gate needed its own fix — it runs blocks in-process with the repo as CWD, so every workflow example left a file. It now uses a throwaway store via
NODUS_WORKFLOW_STORE_ROOTand restores the environment afterwards. That variable also gives a host with a read-only or ephemeral working directory somewhere to put run state.I identified the sources rather than guessing: each run file records its
workflow_name, which maps back totest_cyclic_workflow.py,test_goal_dsl.py,test_workflow_dsl.py,test_workflow_unification.pyand others. Patching them one at a time is the pattern that made #353 recur four times, so the fix sits at the session level.The corrections shipped too.
CLAUDE.mdand the comment intest_task_graph.pyboth said "670+ files cause >2s per sweep". Failures actually start around 400 — 299 files already cost 540 ms against a 500 ms sweep interval. CLAUDE.md now carries the measured table and tells the next reader to check the directory before blaming a flake on timing.One observation logged in the PR rather than swept up:
test_workflow_dsl.py::test_workflow_resume_api_accepts_payload_and_event_metadatafailed once during this work and passed on re-run and in isolation. Same family — an HTTP server test sharing the default runner. Not caused by this change (default_store_root()returns the old value when the variable is unset, asserted by a test; the fixture runs after every test finishes), but it suggests the shared default runner in server tests is worth a look alongside part 2.- added a commit that references this issue
on Aug 15, 2026 Part 2 landed in #385 (
4875b5b), closing this.files list_runs()before → afterexpire_wait_timeouts()before → after300 304 ms → 60 ms — → 48 ms 3,000 3,223 ms → 863 ms 6,600 ms → 579 ms 10,000 13,459 ms → 3,840 ms — → 2,591 ms It was never the parsing, which is why profiling before changing anything was worth the ten minutes. The fix directions listed in this issue — an index file, or querying only non-terminal runs — both assumed the cost was reading and decoding JSON. On 3,000 records, 1.7 s of 4.2 s was in
nt.mkdirand 1.6 s innt.stat; the actual file reads were under a second._runs_root()re-created the store's own directory on every call, and_run_path()calls it once per record — 6,000 mkdir syscalls to list 3,000 runs._load_run_unlocked()did anos.path.existsbefore anopenthat already reports a missing file. Two syscalls where one does, and a race besides.expire_wait_timeouts()re-read every record it had just been handed, for every run rather than only the waiting ones — terminal runs cannot expire, so the sweep was doing two full passes to find, usually, nothing.
An index would have added a consistency problem on top of a store that already has an unexplained persistence race (#376), for less benefit than deleting redundant syscalls.
The count cap is opt-in.
max_terminal_runsprunes oldest-first on write, as suggested, but defaults to off: run records are history a host may rely on, and silently deleting them is not a call this store should make on anyone's behalf. Live runs are never pruned regardless of count.Still linear, so
SQLiteWorkflowStoreremains the answer for genuinely long-lived deployments — the local store's docstring now says so with the numbers rather than "will degrade linearly".Tests assert the syscall behaviour, not elapsed time: a timing assertion would be flaky on shared CI and would not identify which of the three regressed.
- added a commit that references this issue
on Aug 25, 2026 - added 5 commits that reference this issue
on Sep 4, 2026
Summary
LocalWorkflowStore.list_runs()reads and parses every file in.nodus/workflow_framework/runs/on every call, the directory grows withoutbound, and the test suite writes into the repo's own copy of it. So the suite
degrades its own future runs: tests that race a sweep against a deadline start
failing intermittently, pass on re-run, and pass in isolation.
This is already known as a symptom —
CLAUDE.mddocuments it, andtest_task_graph.py::test_worker_death_detected_by_sweeperwas hand-immunizedwith a temp SQLite store and a comment explaining why. What is not recorded is
that the threshold is roughly half what those notes say, and that nothing
stops the directory growing.
Measured
list_runs()against a store of N files (this repo, 2026-08-15):list_runs()Linear, ~1.3 ms per file.
The number that matters:
test_worker_death_detected_by_sweeperdocuments a500 ms sweep interval, and its comment attributes past timeouts to "670+"
accumulated files adding ">2s per scan". At 299 files a single scan already
costs 540 ms — over that interval on its own.
CLAUDE.mdrepeats the 670 figure.Both understate it by about 2x, so failures appear well below the count anyone is
watching for.
Why the directory grows
src/nodus_lang_workflow/runner.py:854defaults the store to a CWD-relativepath:
Anything that runs a workflow with the repo root as CWD writes there — including
pytest tests/. Retention isterminal_max_age_days = 30.0(
store.py:366), which prunes nothing within a session, or a week, of test runs.Most workflow tests are careful and use
tempfileroots. At leasttests/test_server.pyuses the default store (self.server.service.workflow_runner.store),so each suite run adds files that every later run pays for.
Observed consequences
Three intermittent failures during one session on 2026-08-15, all passing on
re-run and in isolation:
test_server.py::test_workflow_runs_endpoint_supports_wait_retry_and_cursor_filters(at 244 files)
docs/guide/workflows-and-tasks.md:51(at 2,434 files)
test_scheduler_fairness.py::test_long_running_task_rotates_with_budget,which
CLAUDE.mdalready lists as flaky under loadrm -rf .nodus/workflow_framework/runsfixed the tree both times it was tried.The directory was back to 299 files after roughly three suite runs.
Why it is worth fixing rather than documenting
A flaky suite teaches people to re-run rather than read failures, which is the
same failure mode as a check that cannot fail: the signal is there and nobody
trusts it. The current state also makes the flakes look unrelated — a server
endpoint test, a doc block, and a scheduler fairness test have nothing obvious in
common.
There is a second, non-test consequence: an embedder or operator using the local
backend in a long-lived process has the same unbounded scan. 10,000 completed
runs is not an unreasonable number for a workflow engine, and that is a 13-second
list_runs().Fix direction
Two halves, separable:
Stop the suite writing to the repo. Point the default store at a temp
root for the whole test session (a
conftest.pyfixture, or the sametemp-SQLite treatment
test_worker_death_detected_by_sweeperalready appliesindividually). A test that does not opt into a store should not leave state in
the working tree.
.nodus/is gitignored, so this is invisible until itbites.
Bound the store's cost. Options, roughly in order of effort:
terminal_max_run_files), pruning oldestterminal runs on write;
list_runs()does not stat-and-parse every entry;(1) removes the flakes. (2) is what makes the local backend honest for anything
long-lived — worth deciding whether the local store is meant for that at all, or
whether SQLite is the answer for a real deployment and the local store should
document itself as a dev convenience.
Reproduction
Run
pytest tests/ -qa few times from the repo root and watch the count climb.Affected versions
v4.1.1 and current
main. The CWD-relative default and the full-directory scanboth predate v4.