MirrorNeuron System Tests

mirrorneuron-system-tests is the installable test orchestrator for a MirrorNeuron workspace. It provides one command, mn_test, for component unit tests, cross-component integration, end-to-end flows, system tests, and the downloaded demo_* blueprint matrix.

Install

Python 3.11 or newer is required. From this directory:

python -m pip install -e .

Component-native dependencies remain owned by their components. For a complete development workspace, set MN_ENV=dev; before each normal run, mn_test refreshes the selected sibling Python projects as editable installs in the same environment. Elixir, Rust, Node, Docker, and OpenShell remain explicit prerequisites for suites that require them. Missing prerequisites are reported as blockers.

export MN_ENV=dev

--list and --dry-run remain read-only. Use --skip-python-refresh when a development run must preserve a deliberately frozen Python environment.

The package exposes only this console command:

mn_test --help

For selection semantics, every CLI option, cluster setup, report handling, and troubleshooting, see the complete usage guide.

Run

From mn-system-tests, the default --mn-set-folder ../ resolves the workspace containing this repository:

mn_test
mn_test --component mn-cli --level unit
mn_test --component mn-api --component mn-python-sdk \
  --level unit --level integration
mn_test --suite contracts --suite blueprints.demo

Filters of the same type are ORed. Different filter types are intersected. For example, the second command selects unit suites belonging to mn-cli; it does not select every unit suite and every mn-cli suite.

Useful inspection and control options:

mn_test --list
mn_test --dry-run
mn_test --offline --suite blueprints.demo
mn_test --blueprints-ref <branch-tag-or-commit> --suite blueprints.demo
mn_test --results-dir /tmp/mn-results --fail-fast

--changed limits component selection to dirty nested Git repositories. Exit codes are 0 for a complete passing selection, 1 for failures or timeouts, 2 for invalid configuration or prerequisite blockers, and 130 for an interrupt.

Levels and safety

• unit: component-local tests and orchestrator logic.
• integration: offline contracts, installer checks, and local live component integration.
• e2e: API, CLI, core runtime, and browser user flows.
• system: benchmarks, performance, scale, cluster, model, storage, and blueprint runtime tests.

MirrorNeuron command suites start an isolated Redis service or validate the configured Redis endpoint before launching Mix, and run the child with MN_ENV=test; missing Redis is reported as a prerequisite blocker instead of allowing Redix retries to consume the ExUnit timeout.

Plain mn_test uses the local-safe-full policy. Suites requiring two-box, gpu, host-secret, or destructive are visible but skipped until explicitly enabled. Repeat --enable when a selected suite requires multiple gates.

Two-box suites default to this workspace's spark target: ssh spark and ~/Projects/mirror-neuron-set. The runner exports that remote configuration to every suite process, so all cluster test modules use the same target. Override both parameters together when using another second box:

mn_test \
  --suite cluster.federation \
  --enable two-box \
  --remote-ssh-command "ssh another-host" \
  --remote-mn-set-folder "~/another/mirror-neuron-set"

The SSH command may contain connection options but no remote shell command. The remote path must be absolute or start with ~/; quote it so the local shell does not expand it. Before running a two-box suite, mn_test connects read-only and verifies the remote workspace. See Running two-box and cluster tests for prerequisites and suite-specific gates.

Ordinary python -m pytest is safe: tests marked live do not start services unless the orchestrator sets RUN_MN_SYSTEM_TESTS=1.

The ungated cluster.federation.local suite is part of normal CI. It starts two isolated Cores, APIs, Context services, standalone Redis instances, and recording LiteLLM/DMR planes. It proves reciprocal CLI/API readiness, single-owner execution, automatic remote-owner selection, Redis-backed cached summaries, partition/reconnection behavior, disconnected Core restart autonomy, and mandatory gateway traversal without requiring Spark.

Performance tests

Run the deterministic injected interface benchmark without starting live services:

MN_ENV=dev mn_test --suite performance

The development refresh installs the selected sibling API, SDK, and CLI projects automatically. The suite records their latency and throughput, plus LLM-stream parser measurements, in performance.txt and performance.json inside the run's result folder. Live API, gRPC, LLM, and Web UI probes are separately opt-in. See Performance tests for a quick local sample, live endpoint configuration, controls, and result interpretation.

Demo blueprint matrix

blueprints.demo maintains a tool-owned cache of MirrorNeuronLab/mn-blueprints, records the resolved Git commit, and discovers all index entries whose IDs start with demo_. It never modifies a user-owned checkout.

Each demo is validated, launched against the real local runtime with --offline --fake-llm --detached and an isolated run ID, polled to completion, checked for declared artifacts and event evidence, copied into the report, and cleaned up by owned run/job ID. Docker and OpenShell are checked only for manifests that declare those drivers. A terminal run may continue using its existing timeout window while detached shared-storage artifacts are copied back; the runner does not move evidence before that copy is complete.

--offline forbids clone/fetch and requires the requested ref to exist in the tool-owned cache.

Results

Every non-dry run writes a new immutable folder under results/runs/<timestamp>-<id>/:

report.html
report.txt
results.json
junit.xml
logs/
services/
pytest/
artifacts/blueprints/

Output streams live to the terminal and to per-step logs. Reports are rewritten after every completed step and in final cleanup, so failures, timeouts, cleanup errors, and interrupts retain partial evidence. Secret-like environment values are redacted from both live output and saved logs.

results.json uses the versioned mn.test.result/v1 schema and includes selection, environment, Git revisions/dirty state, timing, counts, steps, and blueprint revision metadata.

Pytest cache files, coverage output, and the default runtime home are kept in the run-owned pytest/ folder, so normal component checkouts and ~/.mn are not mutated. Suites explicitly marked host-secret retain caller-provided runtime state.

Compatibility

test_all.py remains for one release as a deprecated source compatibility shim. It translates supported historical flags into mn_test arguments and is not installed as a second entry point. New automation should invoke mn_test.

Source layout

src/mn_system_tests/
  blueprints.py       downloaded demo matrix
  catalog.py          declarative suite catalog
  cli.py              public command
  config.py           workspace and run configuration
  models.py           typed suite/result records
  preflight.py        command/module/path checks
  process.py          streaming subprocess execution
  reports.py          JSON, text, HTML, and JUnit reports
  runner.py           orchestration
  services.py         isolated local service lifecycle
  testing.py          reusable test helpers
tests/
  unit/
  integration/
  e2e/
  system/