Open index and federation: artifacts, CIFF export, private crowd signal - #11
Conversation
This is the base of the open-index layer. An open artifact is a published unit (the raw WARC corpus, the CIFF lexical index, the WebGraph link graph, an optional embedding set) produced from one consistent snapshot, so it is exactly what was served. Each artifact is content-addressed, so two parties confirm they hold the same bytes by comparing a SHA-256 digest, and signed by its producing operator with ed25519, so a consumer verifies integrity and provenance before trusting it and a poisoned shard is traceable to who published it. The operator key registry is the trust anchor: an operator it does not know is not trusted, and a key rotation is one replaced entry. The signature covers the artifact identity (kind, snapshot, content address) rather than the bytes, so provenance verifies from the manifest before the bytes are fetched.
CIFF is the lead open artifact: the smallest useful unit a third party can load into an independent engine and verify, which is the M2 interop gate. The exporter walks a segment in CIFF order (terms sorted, postings ascending by docid) and emits the three sections: the header with the collection statistics, the postings lists, and the document records. The Source seam is a pair of iterators, so the exporter streams and never holds the whole index in memory; MemSource is the in-process reference that sorts unordered input into CIFF order and derives the header. The encoding is a deterministic length-prefixed binary with gap-encoded docids that round-trips through Read, so the format is self-contained and tested. The real CIFF protobuf wire format is the same logical structure in a fixed field order; swapping the encoder for the protobuf marshaler is what satisfies the external-engine gate, and the seam does not change.
The leaks made clear that behavioral signals decide ranking, but collecting them the incumbent way means a per-user profile. This collects the same aggregate the late re-ranker needs (good clicks, the last longest click, pogo rate per query and document) while holding no identity. Three parts mirror the Brave Web Discovery design. Squashing collapses a session's repeats for one query and document into a single signal, so a user clicking the same result many times counts once, which is the unsquashed-click spam filter done before submission. Randomized response noises each reported bit on the client under a privacy budget, so what leaves the device is already private and a colluding group can shift the aggregate only so far. The aggregator inverts the noise in expectation, so the published counts are accurate at scale even though no single report is. A test drives twenty thousand noised clients and recovers the planted good-click count to within a few percent. The records carry no identity field; unlinkable transport is assumed, not implemented here.
Federation lets independent operators host different signed shards and have a query fan out across them, without the open peer-to-peer design doc 01 rules out. A partition is just a leaf the root does not own: it answers the same Leaf.Search and carries its signed snapshot id. The gate is the Sybil defense. A partition is queried only if its operator is in the key registry, so its artifacts can be verified, and its reputation clears a floor. An unknown or low-reputation operator is excluded before it can touch a result. The fan-out gives each partition a sub-deadline and merges whatever arrived in time, so a slow or broken federated shard degrades the result rather than breaking the deadline, and the overall wait is bounded so one leaf that ignores its context cannot stall the merge. Reproducible-build verification is the audit. It checks the signature verifies under the operator's key and the independently rebuilt bytes content-address to the same hash the artifact claims. A valid signature over bytes that do not reproduce still fails, because a signature proves who published an artifact, not that it is honest.
| if i > 0 && p.DocID < prev { | ||
| return fmt.Errorf("ciff: term %q postings not sorted by docid", pl.Term) | ||
| } | ||
| if err := putUvarint(w, uint64(p.DocID-prev)); err != nil { // gap encoding |
There was a problem hiding this comment.
The gap encoding is why CIFF stays small, and the writer enforces the precondition it depends on. Postings within a list are sorted ascending by docid, so storing the delta from the previous docid keeps each value tiny and the uvarint short. The writer checks the order as it goes and errors on an out-of-order list rather than emitting a stream that Read would silently decode into wrong absolute ids (the deltas would all be valid uvarints, just meaningless). MemSource sorts on the way in so a caller cannot trip this by accident, and TestGapEncodingHandlesWideDocIDs pins the sparse case where an off-by-one in the delta math would surface.
|
|
||
| func (a *Aggregator) debias(reportedTrue, n int) float64 { | ||
| p := keepProb(a.epsilon) | ||
| est := (float64(reportedTrue) - (1-p)*float64(n)) / (2*p - 1) |
There was a problem hiding this comment.
This line is the whole privacy-versus-utility trade made explicit. Each report is noised on the client, so a single report tells you almost nothing; this inverts the randomized response in aggregate. With n reports and c reported-true, the planted count of true bits is recovered as (c - (1-p)*n) / (2p-1). The denominator is why keepProb must exceed one half (TestKeepProbAboveHalf): at p=0.5 the bit is a coin flip carrying no signal and the estimate diverges. The clamp at zero is because noise can push a near-zero signal slightly negative, which would read as a nonsensical negative click count. TestAggregateRecoversTruthAtScale drives 20k noised clients and lands within a few percent.
| merged = append(merged, pr.results...) | ||
| } | ||
| case <-timer.C: | ||
| // The budget is spent. Serve with what arrived; the rest are dropped. |
There was a problem hiding this comment.
The overall timer is deliberate defense against a misbehaving leaf, not redundant with the per-partition deadline. Each partition already gets its own WithTimeout, so a well-behaved leaf that watches its context returns promptly and the collection loop drains naturally. But a leaf that ignores its context never sends on the channel, and without this timer the loop would block forever waiting for len(trusted) sends. The timer caps the wait at the deadline plus a small slack and serves whatever arrived, which is the good-enough cutoff applied to federation. The abandoned goroutine still finishes and sends into the buffered channel, so it does not leak. TestSearchDropsErroringAndSlowPartitions covers both the error drop and the block-past-deadline drop.
|
On the signing design: the signature covers SigningBytes (kind, snapshot, content address) rather than the artifact bytes themselves. That is the choice that lets a consumer verify provenance from the manifest before it has fetched anything, which matters because the bytes move over the cold-tier transport and may be large. The split is: the signature proves who published this artifact and that the manifest entry is intact, and the content address proves the fetched bytes are the ones the entry names. VerifyRebuild puts both halves together with reproducibility, so a malicious operator cannot publish honest-looking metadata for bytes that do not rebuild. One consequence worth noting for review: this means the signature does not bind to the bytes directly, only transitively through the content hash, which is fine for SHA-256 but is the assumption to keep in mind if the hash is ever weakened. |
|
Scope note on what this PR leaves for later, so the boundaries are explicit. CIFF ships as a faithful-shape reference encoder, not the protobuf wire format; the M2 interop gate (an external Anserini or PISA loading the export) is the protobuf swap behind the same Source seam, and I would rather land a self-contained round-trippable format now than a half-written protobuf that no external tool reads yet. The WARC/WAT/WET and WebGraph exporters are not here either; the artifact and signing machinery they need is, so each is a per-format writer rather than new infrastructure. The crowd signal assumes the unlinkable submission transport (the Brave proxy) rather than implementing it, which is a network protocol orthogonal to the privacy math in this package. And the open API gateway over serve and answer (doc 11.6) lands with the wire serving client, the same place the answer-engine PR deferred the mixer and streaming. All of these are additive behind seams that exist now. |
1 participant
Open index and federation
This is the first differentiator at code level: the index is a set of published, content-addressed, signed artifacts that a third party can download, load into an independent engine, and verify against a reproducible rebuild. Federation rides on the same artifacts, so a partition another operator hosts is just a signed leaf the root does not own. The discipline from doc 01 holds: decentralization serves openness and trust and never costs the relevance and latency bar.
Implements architecture doc 10 (
~/notes/Spec/2050/10_open_index_decentralization.md) and impl doc 11 (~/notes/Spec/2050/implementation/11_open_federation.md).What landed
open/open.goplusopen/sign.go: the artifact vocabulary (ArtifactKind,Artifact), content-addressing (Addressover SHA-256), and ed25519 signing with an operator keyRegistryas the trust anchor.open/ciff/: the CIFF lexical-index exporter, the lead open artifact. A streamingSourceseam (iterators), theMemSourcereference, a deterministic gap-encoded writer, and aReadthat round-trips it.open/crowdsignal/: the privacy-preserving behavioral signal. Client-side squashing and local-differential-privacy randomized response, server-side unbiased aggregation that recovers the true counts at scale.open/federation/: trusted-partition fan-out with a per-partition deadline and a reputation gate, plus reproducible-build verification.Deliberate narrowing
Sourceis an iterator pair so the exporter streams off the segment FST and postings (doc 05) without materializing the index;MemSourceis the in-process reference. The federationLeafis the same shape as the first-partyLeaf.Search(doc 02), so a federated partition is a leaf the root does not own. The production bindings (the real segment source, the wire leaf client) are mechanical swaps behind these seams.Read. The real CIFF protobuf wire format is the same logical structure in a fixed field order. Swapping the encoder for the protobuf marshaler is what satisfies the M2 interop gate (an external Anserini or PISA loads the export); the seam and the section model do not change. Keeping the reference encoder here makes the format self-contained and testable now.VerifyRebuildchecks the signature verifies and the independently rebuilt bytes content-address to the same hash the artifact claims. A valid signature over bytes that do not reproduce still fails, because a signature proves who published an artifact, not that it is honest.Not yet built (later swaps behind these seams)
cmd/publisher, doc 11).serve/andanswer/paths with authentication and quota, which lands with the wire serving client.Tests
go build ./...,go test ./...,go test -race ./open/federation/,go vet, andgolangci-lintare all clean. The open tree covers content-addressing stability, the signing-bytes covering every identity field, sign/verify round-trip and tamper detection, and the registry gating on operator and key; CIFF write/read round-trip with terms and postings coming back sorted, byte-identical determinism, wide sparse docid gap-encoding, bad-magic rejection, and the empty index; crowd-signal squashing, the keep-probability invariant, the de-biased aggregate recovering planted truth across twenty thousand noised reports, and seeded-RNG reproducibility; and federation gate exclusion, trusted-partition merge with the untrusted result kept out, dropping erroring and slow partitions, the K cap, and the rebuild audit (honest pass, mismatch reject, unknown operator reject).