catalog-tests: the catalog test harness, and flowctl raw test - #3417
Open
jgraettinger wants to merge 5 commits into
Open
catalog-tests: the catalog test harness, and flowctl raw test#3417jgraettinger wants to merge 5 commits into
flowctl raw test#3417jgraettinger wants to merge 5 commits into
Conversation
This was referenced Aug 27, 2026
jgraettinger
force-pushed
the
johnny/v2-tests-3-catalog-tests
branch
from
August 30, 2026 17:56
7c143a6 to
db1ce76
Compare
jgraettinger
marked this pull request as draft
August 31, 2026 20:38
jgraettinger
force-pushed
the
johnny/v2-tests-3-catalog-tests
branch
from
August 31, 2026 22:08
db1ce76 to
df38483
Compare
jgraettinger
marked this pull request as ready for review
August 31, 2026 22:11
jgraettinger
force-pushed
the
johnny/v2-tests-3-catalog-tests
branch
from
August 31, 2026 22:19
df38483 to
5dd646c
Compare
Reset asks a derivation shard's connector to reset its internal state to an as-just-initialized condition, mapping onto the connector protocol's existing `derive.Request.Reset`. It is the mechanism the catalog-test harness uses to isolate test cases from one another. Reset is deliberately shard-local — Controller → Shard, with ResetDone coming back Shard → Controller. The leader is not in the path: coordinating a common transaction boundary across shards belongs to the caller, and the test harness already has that property because it drives one transaction per stat() and awaits its commit. ResetDone is nonetheless required. A shard receives transaction messages from its leader but Reset from its controller, so ordering within either stream alone cannot establish that the connector observed the reset before the next transaction's reads.
A derivation shard now answers a controller Reset by queueing a `derive::Request.Reset` to its connector and replying ResetDone. This is handled shard-locally; the leader is not involved.
`Opened.runtime_checkpoint` is how a connector tells the runtime that its checkpoint state is remote-authoritative — that the endpoint, not the recovery log, holds the truth. derive-sqlite reported one unconditionally, including for a `:memory:` database whose "truth" does not outlive the session. Introduce `Database` to decide this once, where the URI is chosen: `Durable` for a file-backed database, whose recorded checkpoint is authoritative and is reported; `Ephemeral` for `:memory:`, which reports none. An explicit `:memory:` from `sqlite_vfs_uri` is now recognized as ephemeral rather than taken at face value as a file path. This is a test-only concern in practice: a production task is always threaded a recorded recovery-log VFS URI by the runtime, and `:memory:` arises only where a local drive deliberately wants a stateless connector. What it buys is that such a drive is no longer forced single-shard by an authority claim the database can't back — which is what lets the catalog-test harness run derive-sqlite multi-shard.
New crate holding what a catalog test *means*, layered above runtime-local and runtime-next. `run_tests` starts a resident session per enabled derivation, then runs every case — sorted by first-step scope, so a run is diffable from one invocation to the next — through the scheduler, with a Reset between cases. Which derivations to start is settled synchronously first, so a malformed catalog fails before any connector boots. Every derivation runs multi-shard, whatever its connector. Sessions thread an empty `Task.sqlite_vfs_uri`, so a derive-sqlite shard runs a session-scoped `:memory:` database and claims no checkpoint authority it couldn't back — which is what leaves the leader free to accept the topology. clock / graph / scheduler carry the scheduling semantics that `go/testing/` worked out — writes cascading through a multi-hop chain, self-cycles reaching a fixed point, read delays against a lazily-advanced synthetic clock, verify steps gated on a forward BFS over pending reads — and follow the Go closely, down to its unit tests, rather than re-deriving them. Two deliberate departures: the graph tracks derivations only, since a test never runs a capture or a materialization; and there is no `ErrAdvanceDisabled`, because synthetic time is a counter local to this crate. Everything else that's non-obvious is written up under "Non-obvious details" in the crate README. Tests run over derive-sqlite and need no containers, covering scheduling, Reset isolation, partition selectors, redaction, collection names nested under one another, read delays, diff rendering, and a run dropped mid-flight. Connector death at startup — while handling Open, or racing the leader's startup commit just after Opened — is covered by `tests/fixtures/dying_connector.py`, since no well-behaved connector produces that fault.
`flowctl raw test --source ...` runs a catalog's tests locally on the runtime-next stack, and CI's catalog-test task now uses it instead of `flowctl-go test`. It sits under `raw` beside `raw preview-next`, which drives the same stack: both are local, developer-facing, and tied to a runtime that is still landing, so neither belongs at the top level yet. `flowctl catalog test` — a remote dry-run publish — is a different command and is unaffected. Output is a line per test, keyed by test name: a failure names the source path and JSON pointer of the step it failed at and indents the rendered diff beneath, and a case the run never reached says so. Then a tally. Exits non-zero if any test failed.
jgraettinger
force-pushed
the
johnny/v2-tests-3-catalog-tests
branch
from
August 31, 2026 22:46
5dd646c to
5106b77
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
Stack 3 of 5, and the core of it: a new
catalog-testscrate holding what a catalog test means, layered aboveruntime-local(#3416) andruntime-next. Plus the one protocol addition it needs, and the CLI that drives it.Five commits:
proto: add derive Reset / ResetDone messages— the mechanism the harness uses to isolate test cases from one another.runtime-next: forward Reset to derive connectors— handled shard-locally; the leader is not involved.derive-sqlite: report no checkpoint for an ephemeral database— a:memory:database is not remote-authoritative, and saying so is what lets the harness run derive-sqlite multi-shard.catalog-tests: the catalog test harness— the crate.flowctl: add the raw test subcommand— and CI'sci:catalog-testtask now uses it instead offlowctl-go test.Workflow steps:
A line per test, keyed by test name. A failure names the source path and JSON pointer of the step it failed at and indents the rendered diff beneath; a case the run never reached says so. Then a tally. Exits non-zero if any test failed.
Documentation links affected:
None user-facing.
crates/catalog-tests/README.mdis new and carries the design.Notes for reviewers:
Suggested reading order:
crates/catalog-tests/README.md— especially "Non-obvious details", which covers the two clock flavors that must never mix, journals keyed bypartition_template.namebecause catalog names may nest, commit signalling through thePublisherseam, ordering guaranteed across store transactions but deliberately not within one, and redaction under an empty salt. Thenrun.rsandscheduler.rs, then the two test files.clock/graph/schedulerfollowgo/testing/closely, down to its unit tests, rather than re-deriving scheduling semantics that were already worked out. Two deliberate departures: the graph tracks derivations only, since a test never runs a capture or a materialization; and there is noErrAdvanceDisabled, because synthetic time is a counter local to this crate rather than something a data plane must serve.On
raw test's placement. It sits underrawbesideraw preview-next, which drives the same stack: both are local, developer-facing, and tied to a runtime that is still landing.flowctl catalog test— a remote dry-run publish — is a different command and is unaffected.Tests run over derive-sqlite and need no containers, covering scheduling, Reset isolation, partition selectors, redaction, nested collection names, read delays, diff rendering, and a run dropped mid-flight. Connector death at startup is covered by
tests/fixtures/dying_connector.py, alocal:connector that dies on command, since no well-behaved connector produces that fault.Stack — each PR is based on the one above it, so review only the top commits of each.
flowctl raw test#3417 — catalog-tests: the harness, andflowctl raw test← you are here