Skip to content

Run tests with nextest so binaries execute in parallel instead of one after another - #883

Closed
WaylandYang wants to merge 1 commit into
devfrom
ci/nextest
Closed

WaylandYang wants to merge 1 commit into
devfrom
ci/nextest

Conversation

@WaylandYang

Copy link
Copy Markdown
Contributor

Follow-up to #879.

Why

cargo test --workspace runs the 140 test binaries one after another, each parallel only within itself. Most binaries hold one to three tests, so the test job spends most of its 86 s run phase waiting for one binary to finish before starting the next. cargo-nextest runs tests across binaries in parallel, one process per test. Locally the run phase drops from 136 s to ~61 s.

What

  • .config/nextest.toml makes explicit the two isolation assumptions cargo test provided implicitly:
    1. utopia-store integration tests never run concurrently with each other (a max-threads = 1 test group). Several assert whole-database state, e.g. a_vector_index_is_built_by_a_job counts queued builds, and the timeline tests take row locks that deadlock when other store tests write at the same time. Under plain nextest these failed deterministically or sporadically; serialized, 8/8 runs were clean.
    2. a_queued_job_wakes_an_idle_worker runs exclusively (threads-required = "num-cpus"). Its three tests share the utopia_jobs NOTIFY channel and serialize through a process-local static Mutex, which does nothing once each test is its own process; other tests enqueueing jobs would also wake it. Under nextest without the rule 2 of 3 tests failed every time; with it 3/3 passed in every run.
  • mcp_tests::Fixture::clean retries remove_dir_all for up to a second. Dropping the tantivy IndexWriter does not wait for its merge threads, which keep writing segment files into search/; deleting the directory immediately raced them about one run in five under nextest (Directory not empty). The other fixtures already ignore the error with let _ =; this one keeps reporting a directory that genuinely cannot be removed.
  • test job: installs the pinned nextest 0.9.146 prebuilt binary from get.nexte.st (seconds; cargo install would compile for minutes), skipped when ~/.cargo/bin from the dev cache already has it. Test runs cargo nextest run --workspace followed by cargo test --workspace --doc, since nextest does not run doctests (there are none today; the step is there so future ones are not silently skipped).
  • CONTRIBUTING.zh-CN.md: one paragraph saying CI uses nextest and where the rules live; cargo test locally still works.

Workflow-level permissions: contents: read untouched.

Verification (local, docker compose Postgres, UTOPIA_TEST_REQUIRE_DB=1)

clippy --workspace --all-targets -D warnings: clean
a_queued_job_wakes_an_idle_worker alone under nextest, 3 runs: 3 passed each
cargo nextest run --workspace, 5 consecutive runs:
  1050 passed, 5 skipped   66.8s
  1050 passed, 5 skipped   60.9s
  1050 passed, 5 skipped   77.1s
  1050 passed, 5 skipped   62.9s
  1050 passed, 5 skipped   60.3s
cargo test --workspace --doc: 0 doctests across 7 crates
cargo test --workspace (same tree): 136s run phase

The 5 skipped are the opt-in MySQL, Trino and live-network RSS tests, skipped under cargo test as well.

Expected on CI: the test job's run phase from ~86 s to ~40 s. Compilation is unchanged; that is the remaining cost and it needs the store integration tests restructured (separate follow-up, see #879 discussion).

🤖 Generated with Claude Code

… after another

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: WaylandYang <145302500+WaylandYang@users.noreply.github.com>
@WaylandYang

Copy link
Copy Markdown
Contributor Author

Closing without merging: on the 4-vCPU runner the nextest run phase measured 157s versus 86s for cargo test (run 35842332445). Per-test process startup dominates there (server tests average 0.73s each under nextest), and the serialized store group holds one of the four slots for 80s. The local 2× win was a 10-core artefact. The real path to a faster test job is fixing the three store-test isolation issues so the store suite can run in one binary in parallel; that is being done separately.

@WaylandYang
WaylandYang deleted the ci/nextest branch September 23, 2026 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant