Conversation
The serving tree is the natural Go fan-out: the root fans to aggregators, each aggregator to leaves, the same shape nested. Scatter fixes that shape per doc 08.1. An errgroup bounded by SetLimit runs the leaf RPCs, results land in a pre-sized slice indexed by shard so there is no channel or mutex on the hot path, and each RPC gets a sub-deadline off the parent context with defer cancel so a straggler's work is freed the moment its budget passes. The load-bearing decision is partial-result tolerance. A leaf error or a missed sub-deadline degrades to a missing shard, not a failed query, because important documents are replicated across leaves so a dropped shard rarely holds the unique best result. Gathered reports which shards answered so the caller can apply a good-enough cutoff. MergeTopK reassembles the global ranking. Each shard returns its results already ranked, so this is a k-way merge across sorted lists through a max-heap rather than a full sort, ordered by the cascade convention with ties broken toward the smaller document id so the page is deterministic regardless of which shards happened to answer. This is the first subsystem with real fan-out concurrency, so it brings in golang.org/x/sync, the errgroup and singleflight the engineering principles (doc 01.3) call for.
A single leaf's 99th-percentile latency becomes the root's median once a query fans out wide, so the tail is the defining problem of the tier and waiting for the slowest leaf is not an option. Hedge is the highest-value technique against it: send to one replica, and if no reply arrives within the expected 95th-percentile latency, send a backup to the next replica and take the first to return, cancelling the losers through the context. The backup fires only for the slow tail, so the extra load is a few percent for a large tail improvement. A failed attempt does not wait out the hedge delay: it frees the next replica immediately, so a fast error recovers as quickly as a slow success would have hedged. With no delay every replica races at once, and with one replica it is a plain call. The function is generic so the same logic hedges a leaf search, an aggregator fan-out, or a vertical call.
Caching carries the load and the warm-path latency; the tail techniques carry the guarantee for the uncached working set, since caching does not solve the tail. Cache is the result-cache seam. The production in-process cache is Ristretto, whose TinyLFU admission deliberately drops some writes, so the seam keeps that swappable; LRUCache is the in-process reference, admitting every Set so a test sees a deterministic hit pattern. The Loader is the piece that matters under load. A popular query expiring from the cache would otherwise let every concurrent request hit the backend at once, which is exactly when the backend can least afford it. Single-flight collapses that stampede to one backend call with the rest waiting on its result, and the wait is done through DoChan so a caller whose own deadline passes returns the context error while the shared load continues for the others. A load error is not cached.
The Coordinator is a serving-tree node: it fans a query to its children, drops the slow tail, and merges the rest. The same type is the root over aggregators and an aggregator over leaves. The good-enough cutoff is not a separate timer but the per-shard sub-deadline (slow shards do not answer in time) plus a minimum-responded floor checked after the gather, so a page built from too few shards returns ErrInsufficientShards rather than being served as if complete. A cancelled query reports the context error instead, since a dropped client is a different failure from a flaky fleet. The tests run the whole tier in-process against fake leaves: the fan-out gathers, tolerates errors, honors the sub-deadline and the parent deadline, and bounds its concurrency; the merge matches a brute-force oracle across every k; hedging fires the backup only when the leader is slow and recovers fast on a leader error; the loader collapses a fifty-way stampede to one call; and the coordinator's cutoff serves three of four shards but refuses one of four. Green under the race detector.
| defer cancel() | ||
| } | ||
| r, err := leaf.Search(cctx, req) | ||
| if err != nil { |
There was a problem hiding this comment.
This return nil is the whole partial-result design in one line, so it is worth being loud about. Returning the error here would make errgroup cancel the sibling RPCs and fail the entire query on one flaky leaf, which is the opposite of what we want at this fan-out width. We swallow it, leave OK[i] false, and let the good-enough cutoff in the coordinator decide whether enough shards came back. The safety argument is doc 05's replication: an important document lives on more than one leaf, so a single missing shard almost never holds the unique best result. A leaf that is wrong rather than slow is a separate problem the health checks handle.
| } | ||
| lastErr = o.err | ||
| // A failure frees us to try the next replica right away. | ||
| if launched < replicas { |
There was a problem hiding this comment.
The fail-fast branch is easy to miss and matters for correctness of the latency story. Without it, a leader that errors at once would still make us sit out the full hedge delay before trying replica 1, so a fast failure would cost more than a slow success. Stopping the timer and launching the next replica immediately means an error is recovered as quickly as the fleet can answer. The pending counter is what lets the loop keep waiting on already-launched attempts after a failure rather than returning early, so we only return the last error once every launched attempt has reported back.
| return v, nil | ||
| }) | ||
| select { | ||
| case <-ctx.Done(): |
There was a problem hiding this comment.
Selecting on ctx.Done here rather than only on the singleflight channel is deliberate. DoChan dedupes the load, so one slow backend call backs many waiters; if a waiter's own deadline passes we want it to return promptly with its context error instead of being held hostage by the shared flight, which keeps one caller's slowness from leaking into another's latency. The flight keeps running for whoever is still waiting, and its result still populates the cache, so the work is not wasted. The trade-off is that the load closure captures the first caller's context, so a production leaf load should derive its own timeout rather than inherit one caller's deadline; worth a follow-up when the real loader lands.
|
On the merge step: MergeTopK runs a k-way merge over the per-shard result lists with a max-heap, which assumes each shard hands back its own results already sorted by the page convention (score desc, then segment, then local id). That holds because a leaf runs WAND and returns a ranked top-k, so the coordinator never re-sorts a whole shard, it only interleaves already-ordered streams and stops at k. The heap holds one cursor per shard, so the cost is the classic O(k log S) rather than O(total log total). If a future leaf ever returned unsorted results this would silently produce the wrong order, so the contract that a Response is pre-sorted is worth stating on the Leaf interface when we write the real RPC client. |
|
Scope note for reviewers, the deliberate narrowing for this tier. Everything here is in-process and tested against the Leaf seam with fakeLeaf, no gRPC or wire format yet, that is the next PR and a mechanical swap behind the same interface. The cache is an LRU reference for correctness; production swaps in Ristretto behind the Cache interface for admission and TinyLFU, the Loader stampede collapse via singleflight is unchanged by that swap. Hedging is wired as a standalone helper rather than threaded into Scatter on purpose, the coordinator's good-enough cutoff already covers the common slow-shard case, and hedging is the heavier hammer we reach for per-replica once the replica topology from doc 05 is real. Numbers like the p95 hedge delay and the good-enough fraction are config, not constants, so they tune per cluster. |
1 participant
Builds the
serve/package: the query-serving tier from architecture doc 08 andimplementation doc 08. The whole tier runs in-process against an in-memory
Leaffake here, so the fan-out shape, the tail-latency techniques, the cutoff,and the caching are all tested without a network, the same seam-plus-reference
pattern the earlier subsystems use.
What landed
serve.Scatteris the fixed fan-out shape of doc 08.1: an errgroup boundedby
SetLimit, results in a pre-sized slice indexed by shard (no channel, nomutex on the hot path), a per-shard sub-deadline off the parent context with
defer cancel, and partial-result tolerance so a dropped shard degrades thequery instead of failing it.
serve.MergeTopKis the k-way merge across the shards' already-rankedlists, max-heap based, with the cascade's deterministic tie-break.
serve.Hedgeis the hedged request: fire a backup replica after the p95delay, take the first success, cancel the losers, recover fast on a leader
error. It is generic over the call's result type.
serve.LRUCache+serve.Loaderare the result cache behind a seam, withsingleflightcollapsing a cache stampede to one backend call and honoringthe caller's context while the shared load runs.
serve.Coordinatorcomposes the three: fan out, apply the good-enoughcutoff (sub-deadline plus a minimum-responded floor), merge the survivors.
This is the first subsystem with real fan-out concurrency, so it adds
golang.org/x/syncfor theerrgroupandsingleflightthe engineeringprinciples (doc 01.3) call for.
Deliberate narrowing
Leafis the seam. The production leaf is a long-lived gRPC client to areplica set (doc 08.6); the tests use an in-process fake. The gRPC client
setup, the connection pooling, and the round-robin resolver are config that
lands when the leaf binary is wired in
cmd/.Cacheis the seam. The production in-process cache is Ristretto (TinyLFUadmission, which drops some writes);
LRUCacheis the deterministic reference.probation, canary requests, micro-partitioning, and selective replication
(doc 08.2) are tracked but not built here; tied requests in particular need a
cross-replica cancel RPC that belongs with the gRPC service.
vertical (doc 08.5) are separate units; the mixer is the same scatter-gather
shape one tier up and lands with the answer engine (doc 09).
plane (doc 10).
Tests
go test ./serve/...covers: scatter gathering all shards, tolerating a leaferror, honoring the per-shard sub-deadline and the parent deadline, and bounding
its concurrency; merge against a brute-force oracle across every k, plus
tie-break and the empty and k-exceeds-total edges; hedge firing the backup only
when the leader is slow, taking the first success, failing fast to the next
replica on an error, the all-fail and single-replica and zero-delay and
context-cancel cases; the LRU get/set/evict/update/zero-capacity behavior, the
loader collapsing a fifty-way stampede to one call, serving from cache, not
caching errors, and respecting the context; and the coordinator merging
children, serving three of four shards under a 0.75 floor, refusing one of four
under a strict floor, and reporting the context error when cancelled. Green
under
go vet,golangci-lint, and the race detector.Consumes the ranking path (#7); feeds the answer engine (doc 09) next.