Activity

1. kriszyp

   added

   enhancementNew feature or requestNew feature or request

   area:searchVector search, HNSW, full-text searchVector search, HNSW, full-text search

   on Jun 11, 2026
2. kriszyp commented on Jun 21, 2026

   

   kriszyp

   on Jun 21, 2026

   MemberAuthor

   More actions

   Thread A hypothesis evaluation (pre-benchmark gate)

   I worked the hypothesis from two sides — the actual HierarchicalNavigableSmallWorld.ts implementation, and the ANN/hubness literature — before anyone spends cycles on the 1M benchmark. Summary: the hypothesis is plausible and sits on a genuine unoccupied gap, but the evidence predicts a modest and asymmetric payoff. The routing half is likely already-free; the semantic half is real but bounded. Recommend reframing the deliverable and running a cheap decisive pre-gate before the full benchmark.

   The hypothesis bundles two claims that the literature treats very differently:

   • R (routing): hub-promoted upper layers preserve or improve recall/latency.
   • S (semantics): hub-promoted upper layers are coarse cluster representatives enabling zoom-out.

   Claim R — routing: likely neutral, with a real tail-recall risk and a low ceiling

   The premise "hubs are better waypoints than random nodes" is well-supported — but it cuts against the value of the intervention, not for it.

   • The hierarchy you'd be modifying is largely redundant in high dimensions. A flat navigable graph matches HNSW on recall and p50/p99 latency for d ≳ 32, using ~38% less memory, across 13 datasets up to 100M vectors. The hierarchy only helps for d < 32. The authors' "Hub Highway Hypothesis": in high-dim, greedy search already routes through emergent high-in-degree hubs — they serve the same function as the explicit layers. ("Down with the Hierarchy: The 'H' in HNSW Stands for 'Hubs'", arXiv:2412.01940, ICML 2025; corroborated by Lin & Zhao 2019, gap vanishes after d≈8.) So explicitly promoting hubs into the hierarchy mostly re-engineers an effect that's already free in the regime our embeddings live in.
   • Theory agrees shortcut/long-range edges only matter asymptotically when d ≪ √log n, and adding edges at random can't improve asymptotic search time (Prokhorenkova & Shekhovtsov, ICML 2020, arXiv:1907.00845).
   • The novel downside is documented: hubness's flip side is antihubs — points in almost no neighbor lists. Over-biasing promotion toward hubs amplifies systematic unreachability of sparse regions. HNSW under churn already develops 3–4% unreachable points / ~3% recall loss (arXiv:2407.07871). This threatens tail recall — the opposite of "holds or improves."
   • Encouraging counterpoint: centrality-as-entry-point is proven to help — NSG/DiskANN/Vamana use the medoid (point nearest the global centroid) as a fixed entry. But that's a single distance-central entry node, not in-degree-based multi-level promotion.

   Claim S — semantics: partially supported, three hard caveats

   This is the stronger half, and the better reason to pursue the work.

   • In-degree hubs really are cluster-central in high-dim multimodal data. The decisive test: on 50 real datasets, in-degree correlates far more strongly with distance to the nearest cluster mean than to the global mean, and the localization strengthens with intrinsic dimensionality (Radovanović et al., "Hubs in Space", JMLR 11, 2010). This directly refutes the naive worry that hubs are just grand-mean artifacts — in clustered data they track local centroids. In-degree is also the right graph signal: kNN-density decouples from centrality in high dim, but in-degree retains it.
   • Caveat 1: hubs ≈ centroids, not = centroids. The strongest hub and the cluster medoid are generally different points, and a minority of hubs straddle boundaries (Tomašev et al., IEEE TKDE 26(3), 2014). Pure hub-prototyping is unstable on real data; prior work falls back to centroid hybrids.
   • Caveat 2: the effect collapses in low dimensions (d=2: hubs sit farther from centroids than average). That's exactly the regime where the HNSW hierarchy is the only thing that matters for routing — so the two desiderata peak in opposite regimes.
   • Caveat 3: an exemplar is not a summary. RAPTOR/GraphRAG earn their zoom-out gains through LLM-synthesized abstraction, and even then the controlled retrieval-only gain (reader held fixed) is modest (~+1.6–2.2 pts QuALITY for RAPTOR; GraphRAG wins comprehensiveness/diversity but explicitly concedes directness to flat vector RAG). Promoting a raw hub record is a structurally weaker form of zoom-out than synthesizing a node — it gives a centroid-adjacent exemplar, not a summary. This is the crux that makes Thread B (summary tree) the more promising thematic-retrieval bet, with Thread A as a free cluster-seed source for it.

   No published work uses graph centrality (in-degree, PageRank, betweenness, k-core) to assign HNSW levels — genuine novelty, but also zero existing validation. The closest non-random scheme is LID-driven promotion (arXiv:2501.13992), which is local geometry, not centrality, and only validated at ~11K points.

   Implementation grounding (current code)

   I had the index read carefully to sanity-check the design sketch. Net: feasible, with two corrections.

   • In-degree cost is cheaper than the proposal worried, for a batch pass. There's no reverse index / in-degree tracking today (Node stores out-edges as node[level] = Connection[]; entry point is a single Symbol.for('entryPoint') value). Per-node in-degree would need a full scan — but the periodic-pass design wants the whole distribution at once, which is a single O(edges) sweep accumulating into a counter. So the proposal's "lists already persisted, no extra structure" instinct holds for the batch pass specifically; it does not hold for any per-write/online variant. Keep this strictly periodic (already in scope).
   • Demotion machinery is reusable; the delete-path reverse-edge cleanup already removes upper-level edges per level and re-elects the entry point with a robust fallback. Promotion is less direct — it must re-run the insert reconnection logic and maintain bidirectional consistency or trigger orphan reindex.
   • No async maintenance hook exists. Table.indexingOperation is the reindex promise; the pass would hook off the indexing-finished schema signal as a separate phase. int8 quantization is orthogonal to topology (no interaction).
   • The cited line numbers have drifted ±15–80 lines (level assignment is at 284/305 not ~271–278; entry election 306–314; delete 452–492; the search descent loop is at 800–810, not 709–718; reindex lifecycle runs to ~1565). The proposal was written against an older revision — worth re-anchoring before implementation.

   Recommendation

   1. Reframe the deliverable. Don't sell Thread A as "better routing" — the literature says that's largely a solved/implicit effect with a low ceiling and a real antihub downside. Sell it as "in-degree-promoted upper layers as a cheap, training-free cluster-exemplar index for zoom-out, and as free cluster seeds for Thread B." That's where the defensible value is.
   2. Run a cheap pre-gate before the 1M benchmark — these three measurements decide go/no-go for a fraction of the cost:
      • (S) Centroid correlation in our embedding space. Build at 100k, compute the layer-0 in-degree distribution (one O(edges) pass — also confirms cost), and correlate in-degree rank with distance-to-nearest-k-means-centroid. If hubs aren't centroid-adjacent here, abandon — the semantic premise fails on this data.
      • (R) Hierarchy contribution ablation. Compare current hierarchical search vs flat (level-0-only) search on the existing graph at 100k/1M. If flat ≈ hierarchical recall (expected in high-dim per arXiv:2412.01940), the routing upside is near-zero before you build any promotion — so success criterion for R is "doesn't regress," not "improves."
      • (reachability) Antihub risk. Measure the unreachable-from-top fraction under random vs a simulated hub-biased level assignment (relabel levels by in-degree on the built graph, no re-wiring needed for a first look). This is the predicted failure mode; quantify it early.
   3. Keep recall parity as the gate, as written — but add tail/sparse-region recall (not just mean Recall@10) explicitly, since antihub loss hides in the tail.

   Confidence: high on the two load-bearing pillars (the JMLR C_cm≫C_dm result; the flat≈hierarchy result), both primary-source verified. The "exemplar ≠ summary" and "regime conflict" points are my synthesis across sources, not claims any single paper makes — well-grounded interpretation rather than established fact.

   Citations: arXiv:1603.09320, 1907.00845, 2412.01940, 2501.13992, 2407.07871, 1707.00143, 1905.10987, 2401.18059, 2404.16130; JMLR 11 (2010); IEEE TKDE 26(3) (2014).

   — Claude Opus 4.8
3. kriszyp commented on Jun 21, 2026

   

   kriszyp

   on Jun 21, 2026

   MemberAuthor

   More actions

   Thread A pre-gate results (10k + 100k)

   Ran the cheap pre-gate before committing to the full 1M benchmark. It tests the hypothesis' three load-bearing premises against Harper's actual HNSW (dist/resources/indexes/HierarchicalNavigableSmallWorld.js, driven over the in-memory store the bundled benchmark uses) on a Gaussian-mixture corpus with known cluster labels + centroids — 768-dim, even clusters, cosine. Two scales: 10k (K=20) and 100k (K=50), 100 queries each.

   Headline: the semantic premise is confirmed and strengthens with scale; the routing premise turns out the opposite of what the high-dim literature predicted — the hierarchy is heavily load-bearing in Harper's implementation, so recall parity is a strict gate, not a formality.

   Results

   Measurement	10k	100k	Reading
   (S) Spearman(in-degree, dist→nearest centroid)	−0.398	−0.629	hubs are cluster-central; strengthens with scale (matches JMLR-2010)
   (S) Spearman(in-degree, dist→global mean)	−0.022	−0.016	~zero → not a grand-mean artifact (C_cm ≫ C_dm confirmed)
   (S) centroid percentile of random upper-layer nodes (lvl≥2)	49.4%	47.2%	random levels are semantically meaningless (as the issue claims)
   (S) centroid percentile of top-1% by in-degree	11.8%	10.0%	centrality promotion would put cluster-central nodes up top
   (skew) antihubs (in-degree 0)	0%	0%	no unreachable points in this regime
   (skew) in-edge share of top 1% of nodes	3.0%	3.8%	not concentrated → low antihub risk on even clusters
   (R) recall@10: hierarchical	93.4%	92.4%
   (R) recall@10: flat (level-0 only)	82.0%	49.4%	hierarchy contributes a LOT, and more at scale
   (R) hier − flat	11.4 pts	43.0 pts	level assignment is load-bearing → parity is a real gate

   Build cost (single-threaded, in-memory): 10k = 45s, 100k = 824s (~13.7 min, 8.24 ms/insert). The full 1M would be ~3–4 hr at this rate — which is why the pre-gate caps at 100k.

   Interpretation per premise

   (S) Semantic premise — SUPPORTED, and the strongest result. In-degree genuinely selects cluster-central records in high-dim clustered data, the correlation strengthens from −0.40→−0.63 as we scale 10k→100k (exactly the JMLR intrinsic-dimensionality prediction), and it tracks the nearest cluster mean, not the global mean (−0.63 vs −0.016). Crucially, this validates both halves of the issue's framing at once: random upper layers sit at the ~50th centroid percentile (meaningless), while in-degree-promoted nodes sit at the ~10th (very central). In-degree is a cheap, training-free signal that does pick good cluster exemplars. This is where Thread A's value is, and it's real.

   (R) Routing premise — must correct my earlier read. In my literature evaluation above I leaned on "flat ≈ hierarchical in high dim, so the routing benefit is already free / low ceiling." That does not hold for Harper's implementation on this data — flat level-0 search collapses to 49.4% recall at 100k vs 92.4% hierarchical (a 43-pt gap that widens with scale). The hierarchy is doing major routing work here. Implication for the hypothesis: the routing claim "recall holds or improves" is unproven and the regression risk is real — level assignment is load-bearing, so any centrality reassignment must clear strict recall parity (incl. tail recall), not just match the mean. Caveat: my flat baseline is a weak one — single fixed entry point, ef=50, same graph. A purpose-built flat-NSW with multi-entry seeding / higher ef would close much of that gap (the "Down with the Hierarchy" result), so 43 pts is an upper bound on the hierarchy's irreplaceable value, not a measurement of it. But the direction is clear: don't treat level reassignment as free.

   (skew) Antihub risk — benign here, but this is the optimistic case. Zero antihubs, top-1% holds only 3.8% of in-edges. The predicted sparse-region-unreachability failure mode doesn't appear — but my corpus is even-sized, well-separated Gaussians. Skewed cluster sizes, outliers, and post-deletion churn (the issue's own benchmark axis) are where antihubs would emerge, and those aren't tested yet.

   Verdict & recommendation

   Proceed to a prototype — but framed as the issue's reframed goal, not "better routing." The pre-gate clears the semantic premise decisively (the whole point of zoom-out) and quantifies the routing constraint. Concretely:

   1. Green-light Thread A as a training-free cluster-exemplar / zoom-out index (and as free cluster seeds for Thread B). The in-degree→centrality signal is confirmed and scale-robust.
   2. Treat routing as a strict do-no-harm gate, not a hoped-for win. Add tail / per-cluster recall to the parity criterion — antihub loss hides in the tail, which mean Recall@10 won't show.
   3. Before the 1M benchmark, two cheap additions decide remaining risk: (a) re-run on skewed cluster sizes + outliers + post-delete churn to stress the antihub mode that the even-cluster corpus suppresses; (b) ef sweep on the flat baseline to bound how much of the 43-pt gap is the hierarchy vs my weak single-entry flat search.
   4. The real prototype (build a centrality-promoted graph with rewiring via the periodic pass and measure recall parity + zoom-out coverage) is the next step — this pre-gate deliberately measured premises on the as-built random graph, not a promoted one.

   Reproduction: ~/dev/scripts/hnsw-thread-a-pregate.js [N] [dims] [K] [queries] (standalone, requires the built dist). Raw 100k log available on request.

   Confidence: high on (S) — clean, scale-consistent, replicates the established hubness/centrality result on Harper's own graph. Medium on the (R) magnitude — directionally solid (hierarchy is load-bearing) but the 43-pt number is inflated by a deliberately weak flat baseline.

   — Claude Opus 4.8
4. kriszyp commented on Jul 7, 2026

   

   kriszyp

   on Jul 7, 2026

   MemberAuthor

   More actions

   Design review (Fable)

   A deeper design pass on this issue. Full working notes are extensive; the load-bearing conclusions:

   The real target is a new capability — zoom-out / thematic retrieval — not recall or latency. Worth being explicit about that, because the pre-gate downgraded routing to a "do no harm" gate for good reason: Harper's hierarchy is heavily load-bearing (92.4% vs 49.4% flat recall@10 at 100k), partly because our descent is non-canonical — a full ef-wide searchLayer at every level (HierarchicalNavigableSmallWorld.ts:801-810), not the canonical greedy single-path through the upper layers. So an in-place level reassignment perturbs a structure that is doing more work than a textbook HNSW's upper layers do.

   Semantic premise holds but the intervention is unmeasured. In-degree ↔ centroid distance ≈ −0.63 at 100k (central nodes ~0 vs global mean) — but that was measured on friendly even Gaussians and on the as-built random graph. The promote-and-rewire intervention itself has not been measured.

   Invariants any promotion pass must preserve: bidirectional edge symmetry (checkSymmetry:878), no level-0 orphans (:499-583, :939-942), single valid entry point + atomic swap (:452-492), updates preserve level (:305), indexing runs synchronously inside the record write txn (Table.ts:4414), and graphs are node-local (not replicated) and currently un-versioned (interplay with VT-caching).

   Options considered

   1. In-place centrality level reassignment (the issue's Thread A) — answers the research question; carries the routing risk above + rewiring churn cost.
   2. Exemplar overlay — same in-degree sweep, but zero graph mutation and therefore zero recall risk; delivers the confirmed semantic value as a separate access path.
   3. RAPTOR-style summary tree (Thread B) — no core changes, highest thematic quality (an exemplar ≠ a summary), LLM build cost.
   4. Medoid-entry / ε-mixed bias — proven, but doesn't deliver zoom-out on its own.

   Recommendation: prototype Option 1 as the research vehicle, but build it behind a clean seam — a shared computeExemplarTiers() feeding either applyAsLevels() or applyAsOverlay() — so Option 2 is a drop-in productization fallback if level-parity fails. Implement the level zoom-out query first (trivial: the sort object already flows whole through Table.ts:2741 → HNSW search()). Green-light Thread B independently.

   Validation gates: mean recall@10 within 1% and per-cluster-min / p5 tail recall within ~2–3%; skewed-cluster + churn corpora; connectivity BFS + zero-asymmetry after every pass; crash-mid-pass validity; pass churn bounded with hysteresis.

   Open questions: accept the overlay fallback as the productization path? API granularity (16×/level vs fractional tiers)? where does the promotion pass trigger? inter-pass insert policy (random draw by default)? versioning sequencing vs the un-versioned-node reality? 1M benchmark budget (~3–4h build)?

   — Design review by Claude (Fable 5), commissioned by Kris.
5. kriszyp

   added a parent issue on Jul 7, 2026

   • [Epic] Vector/HNSW roadmap (research) #1658
6. kriszyp commented on Jul 12, 2026

   

   kriszyp

   on Jul 12, 2026

   MemberAuthor

   More actions

   Thread B pre-gate results — RAPTOR-style summary tree, end-to-end on real Harper pieces

   Ran the Thread B evaluation the same way Thread A was pre-gated: cheaply, against Harper's actual HNSW (dist/resources/indexes/HierarchicalNavigableSmallWorld.js), before committing to the full reference component. Unlike Thread A's synthetic Gaussians, this one is end-to-end real: a real corpus with real topical structure, real text embeddings, and real LLM-generated summaries.

   Headline: the collapsed-tree premise holds on Harper's own index. Thematic topic-coverage@10 goes 15.8% → 38.5% (2.4×) at the same token budget, factoid recall@5/@10 is completely unregressed, and the two-table variant is metric-for-metric identical to the single-table one — so the pattern needs no core changes at all. The incremental-maintenance scheme (Harper's differentiator vs batch RAPTOR) costs ~4 LLM calls for a realistic single-file update vs 45 for a full rebuild.

   Setup

   • Corpus: Harper's own docs (learn/ + reference/ + fabric/): 724 chunks (~350 tokens, heading-aware) from 106 files across 27 topic areas. The reference/<topic>/ directory structure supplies topic labels — thematic ground truth that is independent of the clustering/summaries.
   • Embeddings: local nomic-embed-text (768-dim, cosine, document/query prefixes).
   • Tree: spherical k-means (k-means++, deterministic seed) → 41 level-1 clusters (median 15 chunks, tiny clusters merged); LLM-summarized (~130 words each, standalone informative prose); recursed → 4 level-2 domain summaries. Cluster→topic purity 0.66 — clusters are semantically real but deliberately not identical to the directory taxonomy.
   • Queries, authored blind: 53 factoid queries (each targeting one specific chunk; the writer saw only that chunk) and 38 thematic queries (each targeting a topic area; the writer saw only topic names + file titles — never the summaries). Ground truth is therefore uncontaminated by the tree build.
   • Variants, all on the real HNSW (default parameters, auto-scaled ef):
     • FLAT — chunks only (baseline)
     • COLLAPSED — chunks + all 45 summaries in one graph (the issue's shared-table design)
     • TWOGRAPH — separate chunk and summary graphs, results merged by distance (the two-table design; distances are comparable under the same metric)

   Factoid results (target = one specific chunk)

   variant	recall@1	recall@5	recall@10	MRR@10	summaries in top-10
   FLAT	56.6%	77.4%	81.1%	0.656	0
   COLLAPSED	50.9%	77.4%	81.1%	0.621	1.42
   TWOGRAPH	50.9%	77.4%	81.1%	0.621	1.42

   The summary tier costs 5.7 pts at rank 1 and nothing at rank 5/10 — the metrics that matter for RAG context assembly. I inspected all three rank-1 displacements: in every case the displacing result was a topically apt summary (e.g. a blob-flush question displaced by the blob-storage cluster summary; one was the target's own cluster summary), with the target chunk still in top-5. For a reader consuming top-k context this is benign; arguably the summary even adds framing.

   Thematic results (target = a docs topic area)

   Coverage@10 = fraction of the target topic's chunks covered by the top-10 results, where a chunk covers itself and a retrieved summary covers its on-topic member chunks (transitively for level-2). This is summary-level coverage — it assumes the summary usefully represents its members, which is RAPTOR's premise (validated by their QA evals); this eval measures the retrieval layer.

   variant	coverage@10	precision@10	words/top-10
   FLAT	15.8%	47.9%	1595
   COLLAPSED	38.5%	49.7%	1537
   TWOGRAPH	38.5%	49.7%	1537

   2.4× topic coverage at a slightly lower token budget, with no precision cost. Robust to the summary-relevance threshold (37.9% at a stricter criterion). This is the zoom-out capability the issue is after, quantified: one retrieved summary stands in for a whole corpus region that flat top-10 physically cannot cover.

   The two-table design is free. TWOGRAPH matches COLLAPSED on every metric. So the pattern works today with Chunk and Summary as separate tables and an app-side merge of two searches — no engine-level union ANN required (there is a live TODO: Support index-assisted sorts of unions at Table.ts:2783, but it is not a blocker).

   Composition with Thread A: summaries become hubs

   In the collapsed graph, summary nodes' in-degree strongly exceeds chunks' once insertion order is controlled (naively appending summaries last hides this — in-degree accrues from later inserts):

   insertion order	summary mean in-degree	chunks	summary in-degree percentile
   summaries last	12.8	18.3	28.8%
   interleaved	24.7	17.5	73.2%
   summaries first	34.6	16.9	92.0%

   This is the Thread A connection made concrete: summary embeddings approximate cluster centroids, and Thread A's pre-gate showed in-degree tracks centroid-proximity (ρ = −0.63 at 100k). In a shared graph, summaries are exactly the nodes a centrality-promotion pass would elevate — resolving Thread A's core weakness (an exemplar ≠ a summary) by giving the promotion pass actual summaries to promote. The threads don't just compose; the shared-table variant makes Thread A's zoom-out layers better than anything raw-record promotion could produce.

   Incremental maintenance (the "living tree" vs batch RAPTOR)

   Public RAPTOR is batch/static. The Harper pattern's differentiator is incremental: assign new/changed chunks to the nearest existing centroid, dirty-mark touched clusters, re-summarize only those (dirty L1 propagates to its L2 parent). Simulated on the real tree (200 trials/config, no LLM needed):

   arrival model	update size	dirty L1	dirty L2	LLM calls	tree untouched
   uniform (worst case)	1 chunk	1.0	1.0	2.0	96%
   uniform	24 chunks	17.4	4.0	21.4	53%
   uniform	72 chunks (10% churn)	31.9	4.0	35.9	20%
   by-file (realistic)	1 file (~7 chunks)	2.3	1.6	3.9	91%
   by-file	3 files (~21 chunks)	6.3	3.1	9.4	79%

   vs 45 calls for a full batch rebuild. Real corpus updates are localized — an edited file's chunks land in few clusters (p50 0.33 clusters/chunk) — so the realistic cost of keeping the tree live is ~4 summary calls per file edit, degrading gracefully toward (never past) batch cost under pathological uniform churn. Caveats: the sim doesn't model centroid drift (a periodic re-cluster pass with hysteresis is still needed at some churn threshold), and L2 dirty propagation saturates quickly at this tree size — deeper trees amortize better.

   Feasibility grounding (current main)

   The issue's assumptions have drifted — in Thread B's favor. The big one: scope.models is no longer "once available" — it exists and is complete. models.generate()/generateStream() with OpenAI/Anthropic/Ollama/Bedrock backends (resources/models/Models.ts, exposed at components/Scope.ts:94). The build loop's summarization step is a plain generate() call from a component today. Likewise @embed is fully implemented on the write path: pre-commit embedding hook (resources/models/embedHook.ts → Table.ts:2536-2555), auto-created HNSW index (graphql.ts:262), reindex on model change. Summary rows inserted by the build loop get embedded and indexed for free, exactly as the issue's schema sketch assumed.

   Frictions encountered (the issue asked for these; none is a blocker — filing as follow-ups):

   1. No bulk vector-scan API. Clustering must iterate full records and decode each just to read one attribute (primaryStore.getRange); the HNSW index store is rangeable but holds int8-quantized copies (default on), not the true vectors. At 1M×768-dim this answers the issue's open question: yes, a vector-scan API is worth having.
   2. No query-time embedding surface. HNSW requires a caller-supplied target vector; retrieval must call models.embed(queryText) itself before searching. A search-integrated text→vector path would make the whole pattern declarative.
   3. No scheduler primitive for the build loop. Components self-schedule via setInterval in startOnMainThread (the same pattern core's internal sweeps use). Workable; a first-class scheduled-job API would be cleaner.
   4. No union ANN (Table.ts:2783 TODO) — demonstrated non-blocking above, but noting for completeness.

   Verdict & next steps

   Green-light Thread B. All four acceptance-relevant claims survive the pre-gate: thematic zoom-out gain is large (2.4×) and threshold-robust; factoid retrieval is unregressed where it matters (top-5/top-10); the pattern is purely additive and buildable today with zero core changes (two-table variant proven equivalent); incremental maintenance is an order of magnitude cheaper than batch rebuild under realistic update locality.

   Honest limits of this pre-gate: one corpus (724 chunks, docs domain), one embedder, retrieval-level metrics only (no end-to-end QA answer scoring), LLM-authored queries (though authored blind to the tree). The natural next steps:

   1. Reference component implementing the build loop (Chunk/Summary tables + @embed + models.generate + dirty-set incremental maintenance), per the issue's deliverable.
   2. Scale test on a larger, non-docs corpus + an end-to-end QA eval (reader held fixed) to confirm the retrieval gain survives answer generation.
   3. Prototype the Thread A × B composition: shared graph with summaries as promotion candidates — the in-degree evidence above suggests centrality promotion would select summary nodes "for free."

   Reproduction (all in ~/dev/scripts, standalone against harper dist): hnsw-thread-b-corpus.js (chunk+embed) → hnsw-thread-b-cluster.js (k-means, --level 2 --k 7 to recurse) → hnsw-thread-b-export.js / LLM summaries / hnsw-thread-b-collect.js → hnsw-thread-b-eval.js (variants+metrics) → hnsw-thread-b-incremental.js (maintenance sim). Deterministic seeds; raw outputs available on request.

   Confidence: high on the directional results (coverage gain, no-regression at k≥5, two-table equivalence, incremental cost) — they're consistent across variants and robust to the analysis knobs I varied. Medium on magnitudes generalizing (single small corpus; summary-level coverage credits the summarizer's faithfulness, which retrieval metrics can't audit).

   — Claude (Fable 5)
7. kriszyp commented on Jul 12, 2026

   

   kriszyp

   on Jul 12, 2026

   MemberAuthor

   More actions

   Thread B addendum — reference component built + demoed end-to-end; vector-scan friction re-measured

   Two follow-ups to the pre-gate results above.

   1. Reference component — built and running end-to-end

   Turned the pattern into an actual Harper application (schema + resources + a small pure-JS clustering/summarization lib) and ran it against a live Harper 5.2 instance with local Ollama backends. Zero core changes — it's a plain app component. The full loop works:

   • Chunk writes are embedded + HNSW-indexed by @embed on the write path (no app code).
   • A leader-elected maintenance pass clusters the chunk embeddings (spherical k-means), LLM-summarizes each cluster via models.generate, and inserts Summary rows — themselves auto-embedded/indexed by @embed — recursing to a second level.
   • A RagSearch resource embeds the query (models.embed) and merges Chunk + Summary vector searches by exact cosine distance (the two-table design the pre-gate showed is equivalent to a shared table).

   Live behaviour matched the pre-gate: a thematic query ("how does replication work?") returns the replication summary ranked #1 above every chunk; the broadest query surfaces a level-2 domain summary in the top-3; factoid queries show no summary pollution in the top results.

   The "living tree" (the differentiator vs batch RAPTOR) works as designed. A table subscription assigns new/updated chunks to the nearest cluster centroid and dirty-marks the cluster; the pass re-summarizes only dirty clusters, propagating dirtiness to parents. Injecting four new "5.3 replication" chunks into a built 724-chunk tree triggered exactly one level-1 re-summarization (which correctly absorbed the new material into the cluster summary) plus its parent — not a rebuild. That's the ~4-calls-vs-full-rebuild economics from the maintenance simulation, observed for real.

   One new friction surfaced while running it, worth a follow-up: the Ollama model backend forwards only temperature/maxTokens, so it can't set num_ctx. Ollama's 2048-token default then silently truncates long summarization prompts — which presents as low-quality output (meta-language summaries) only on large clusters, not as an error. Any component doing long-context generation through that backend hits this. Workaround is a model variant with num_ctx baked in; the real fix is threading a context/options passthrough into the backend.

   2. Bulk vector access — re-measured; the friction is API-surface, not fidelity

   In the pre-gate I flagged "no bulk vector-scan API" and noted the HNSW index store holds int8-quantized copies rather than the full-precision vectors. Kris rightly pushed back: the int8 nodes carry a per-vector scale, so the vectors are reconstructable (bin × scale), just at 8-bit precision — so is that precision actually a problem for clustering?

   Measured it. Reconstructing all 724 doc vectors from int8 and re-running the identical spherical k-means:

   • Mean relative L2 reconstruction error: ~1.0%.
   • Cluster agreement vs float32 (permutation-invariant Rand index): 98.6%; cluster→topic purity identical (0.659 vs 0.666).
   • For calibration, two float32 runs from different seeds agree at only 30.5% pairwise-co-membership (Jaccard) — i.e. k-means is far more sensitive to its own initialization than to int8 reconstruction error.

   So clustering off the reconstructed int8 vectors is fine, and the friction reduces to "there's no supported API to scan the index store's vectors back out" — a surfacing gap, not a data-fidelity gap. That makes a read-only vector-scan API a lighter lift than I implied: it can hand back reconstructed int8 vectors (or full-precision for quantization: "none" indexes) without needing a new full-precision store. Filing accordingly.

   — Claude (Opus 4.8)
8. kriszyp commented on Jul 13, 2026

   

   kriszyp

   on Jul 13, 2026

   MemberAuthor

   More actions

   Thread A × B composition + end-to-end QA — and a reference repo

   Closing the loop on the two remaining questions: does the collapsed-tree gain survive end-to-end answer generation, and do Threads A and B actually compose?

   Reference component (from the addendum above) is now a standalone repo: https://github.com/HarperFast/summary-tree — schema + resources + a small pure-JS clustering/summarization lib with unit tests. Pure Harper app, no core changes.

   End-to-end QA: the retrieval gain survives answer generation

   The pre-gate measured retrieval quality; this measures whether that turns into better answers with the reader held fixed. New corpus, deliberately different from the docs pre-gate: 1,410 chunks from 42 Wikipedia astronomy articles across 5 categories (~2× the pre-gate, non-docs). Built the tree (66 level-1 + 5 level-2 summaries), then wrote 46 factoid + 25 thematic questions with reference answers, authored blind to the tree.

   For each question the same reader model answers twice — once from flat (chunks-only) top-K, once from collapsed (chunks + summaries, merged by cosine) top-K, same K. Answers were scored by blind pairwise judges (order randomized, reference answer provided):

   question type	collapsed wins	flat wins	tie	collapsed win-rate of decisive
   factoid (n=46)	6	8	32	42.9% — statistical wash, no regression
   thematic (n=25)	13	6	6	68.4%

   Factoid is a 70%-tie wash (flat is fine when the answer lives in one chunk); thematic — where the answer must synthesize across articles — collapsed wins decisively. Position bias checked clean (raw A/B picks 15/18). The mechanism, from inspecting cases: for "which moons show evidence of subsurface oceans?", flat retrieved only Io's ocean and hedged; collapsed pulled four cluster summaries and correctly named Europa, Ganymede, and Enceladus. Summaries reached the collapsed top-K in 23/25 thematic questions but only 32/46 factoid — that asymmetry is the whole result.

   So the pre-gate's retrieval-level finding (flat ≈ collapsed on factoid, collapsed ≫ flat on thematic) holds end-to-end, on a second corpus and a fixed reader.

   Thread A × B compose: centrality promotion selects the summaries

   The open question from Thread A's pre-gate was that a promoted raw record is an exemplar, not a summary. Thread B builds real summaries and (pre-gate finding) they sit at the 73rd–92nd in-degree percentile in the shared graph. So: if you apply Thread A's centrality level-assignment to the shared (chunk+summary) graph, do the promoted upper tiers fill with summaries?

   Measured it on the real HNSW (in-degree averaged over 6 build orders, then the standard exponential level schedule filled from the top of the in-degree ranking — the "relabel, no re-wire" first look Thread A's design review recommended):

   • A summary is 8.0× more likely to be promoted to level ≥ 1 than a chunk (35.6% vs 4.4%); summaries are 5.7–6.8× their 5.9% base rate in the top in-degree tiers.
   • All four level-2 root summaries land at the 82nd–98th centrality percentile.
   • Zoom-out coverage: the centrality-promoted top-K covers ~2× more distinct topics than a random-level top-K (74% vs 37% at K=16).

   So the threads compose as hypothesized — run Thread A's promotion pass on a graph that contains Thread B's summaries and the promoted hierarchy fills with summaries, giving Thread A real summaries to promote instead of raw exemplars.

   Two honest caveats. (1) The single highest-in-degree node is still a well-connected chunk, not a summary — the L2 roots are top-2-18% but not #1, so making the graph's entry point a summary needs an explicit tie-break, not centrality alone. (2) The explicit summary tree still dominates both on zoom-out (100% topic coverage from just the 4 L2 nodes), which is the argument for Thread B's designed tree over Thread A's implicit promotion. Recall-parity under an actual promote-and-rewire pass remains the one unbuilt piece — though the pre-gate's two-table ≡ shared-table result already shows adding summaries doesn't hurt retrieval.

   Where this leaves the acceptance list

   • Thread A prototype/benchmark — evaluated (pre-gate + design review above); full promote-and-rewire recall-parity run is the remaining Thread A build.
   • Zoom-out level query — implemented in the reference component (mode=summaries browses the tree).
   • Thread B reference implementation + retrieval-quality evaluation — done (repo + pre-gate + this end-to-end QA).
   • Default-path behavior — untouched; the whole pattern is additive schema + an app component.

   Reproduction scripts (standalone, drive Harper's real dist HNSW): ~/dev/scripts/hnsw-thread-ab-promotion.js (composition) and hnsw-qa-{corpus,eval,judge-prep,tally}.js (end-to-end QA). Deterministic seeds; raw outputs on request.

   — Claude (Opus 4.8)
9. kriszyp

   set

   Priority

   to

   P2

   on Aug 7, 2026
10. kriszyp

    added the

    Task

    issue type on Aug 7, 2026