Quorum Overlap Lab

An interactive, deterministic lab for a distributed-database failure that looks healthy from the application boundary: a write succeeds, a read succeeds, and the read still returns an older version.

The default experiment uses five replicas with R=2 and W=2. A topology shift biases reads toward replicas that did not acknowledge the latest write. Both quorums remain available and every operation returns success, but the read and write sets need not intersect.

Default finding: 227,520 stale read responses hide behind 100% application success during a six-minute topology incident.

Explore telemetry.sh to correlate the traces, structured logs, and replica/version dimensions that expose this gap.

Why this lab exists

Health checks and status-code metrics answer whether the system replied. They do not prove that the replicas which answered a read observed the latest acknowledged mutation.

For replication factor N, read quorum R, and write quorum W, the familiar intersection condition is:

R + W > N

If that inequality is false, a successful write quorum and a successful read quorum can be disjoint. During the default incident:

write v184 → A + B       ✓ W=2 acknowledged
read request → C + D     ✓ R=2 answered v183
intersection             ∅

This lab is intentionally different from a replica-lag demo. The central failure is quorum geometry and missing version evidence, not just the existence of a lagging replica.

Run it

You need the .NET 10 SDK.

make run

Open http://127.0.0.1:8080.

The browser sends parameter changes to the native Visual Basic model. No framework, database, package registry dependency, or external service is needed at runtime.

Run the command-line model:

make model

Or run the production container:

docker build -t quorum-overlap-lab .
docker run --rm -p 8080:8080 quorum-overlap-lab

The four strategies

Strategy	Consistency outcome	Cost made visible
Flexible R=2 / W=2	May return stale data	100% success conceals stale responses
Strict majority	Forces intersection	More replica contacts and higher latency
Session-pinned reads	Reads from a write acknowledger	Load concentrates on a small replica set
Client version floor	Rejects an observed version below the session floor	Extra replica contacts and retry latency

All four are modeled across the same per-second workload. The goal is not to declare one universal winner; it is to make each consistency trade measurable.

What telemetry.sh reveals

Status codes alone cannot distinguish a correct read from a stale one. The lab emits the join dimensions needed to reconstruct the proof:

trace_id
session_id
key_hash
coordinator_id
replica_set
quorum_member_ids
consistency_level
mutation_version
observed_version
topology_epoch

A useful investigation starts by joining the write and read paths on trace_id/session_id, comparing mutation_version to observed_version, and then grouping by topology_epoch and quorum_member_ids. That connects a customer-visible stale value to the exact membership decision that produced it.

The interactive page includes a representative cross-signal query.

Model

The simulation is analytic and deterministic so every run is reproducible:

• 600 seconds at 1,000 operations/second
• 20% writes and 80% reads
• five replicas with R=2, W=2
• topology shift at 120 seconds
• 70% of reads biased toward the stale-side replica set
• anti-entropy repair at 480 seconds
• fresh replicas respond in 12 ms; stale replicas respond in 6 ms

Before the topology shift, reads are modeled as fresh. During the incident, the unbiased non-intersection probability is:

C(N - W, R) / C(N, R)
C(3, 2) / C(5, 2) = 30%

With a 70% stale-zone routing bias, the incident probability becomes:

0.70 × 100% + 0.30 × 30% = 79%

The model deliberately gives stale replicas lower latency. That reproduces an important diagnostic trap: a stale path can improve latency while degrading correctness.

Parameters are normalized to safe bounds by the server. You can also supply command-line model overrides as uppercase environment variables:

READ_QUORUM=3 WRITE_QUORUM=3 make model

API

POST /api/simulate accepts a JSON object containing any model parameters:

curl http://127.0.0.1:8080/api/simulate \
  -H 'content-type: application/json' \
  -d '{"replication_factor":5,"read_quorum":3,"write_quorum":3}'

The response contains normalized parameters, traffic facts, strategy summaries, per-second timelines, telemetry join fields, and the headline finding. Unknown fields are ignored; known fields must be finite JSON numbers. Request bodies are limited to 64 KiB.

Project layout

Program.vb                   model + dependency-free HTTP server
public/                      interactive topology and timeline explorer
tests/model_test.py          deterministic behavioral checks
tests/http-test.sh           API, security-header, and frontend checks
tests/container-test.sh      production image smoke test
.github/workflows/ci.yml     native and container CI gates

Verify

make check
make container-test

The first target compiles with Option Strict On and warnings as errors, then tests the model, server, API validation, security headers, JavaScript syntax, and responsive CSS. The second builds and probes the final non-root container.

Further reading

• Apache Cassandra: Dynamo architecture explains how W + R > RF provides quorum intersection.
• Apache Cassandra: guarantees distinguishes the guarantees surrounding acknowledged writes and reads.

The lab is an educational model, not a benchmark or a claim about one database's implementation.

License

MIT — see LICENSE.