Research: cut write-path LLM calls — relation inference is O(N) per atom

[nikita-rulenko]

Research ticket — no code yet. Find the cheapest way to build a memory without weakening edge quality or pushing cost onto the (LLM-free) read path.

The cost today

Per add_memory that extracts N atoms, the write path makes:

Stage	LLM calls	Notes
Extraction / atomization	1	one call over the whole input (+1 retry only on bad JSON)
Dedup decision	0–1	already batched (decide_batch); gated exact/cosine dups skip the LLM
infer_relations	0–N	one call PER ATOM (Phase D, concurrent but still N calls; each retried before #95)

So total ≈ 1 + 1 + N. A 5-fact write ≈ 7 calls. The only term that scales with content is relation inference — that's the target.

Constraints that shape the fix:

• Reads are already LLM-free and edges are consumed only at read time — nothing on the write path needs the edges present the instant the write returns.
• Canon: the writer pays so the reader flies — do not move cost to reads.
• Golden reasoning_extraction causal contract (BECAUSE/IMPLIES on fresh writes) must still hold.

Three convergent levers (externally researched, sourced)

Lever 1 — Batch relation inference (N → 1)

One structured-output call over all new atoms × their candidate neighbours, numeric-indexed, chunked ~10–15 pairs/call, strict schema returning [{atom_idx, candidate_id, relation_type|null}].

• BatchPrompt at batch=32 used 15.7% of the calls / 18–30% of tokens with accuracy flat-to-better; sweet spot 3–15 items/batch; reliability needs explicit indices + (optionally) permutation-vote + semantic grouping.
• Strict structured output beats JSON-mode (JSON-mode can loop / emit an array under the wrong key — literally our infer_relations: array prompt under json_object mode → systematic 0-relations retry + silently lost edges (weak-model hostile) #95 bug).
• Sources: BatchPrompt openreview.net/forum?id=Agyicd577r · OBP vldb.org/pvldb/vol18/p2172-zhang.pdf · structured-output agenta.ai/blog/the-guide-to-structured-outputs-and-function-calling-with-llms
• Impact: O(N) → O(1). Keeps inference synchronous. Lowest-risk, highest-leverage.

Lever 2 — Route cheap edge types off the LLM entirely

• RELATES_TO ← cosine similarity threshold (top-k gate; can't tell support from contradiction — negations embed close).
• IS_A / PART_OF ← Hearst/connective pattern rules (high precision) — we already have a structural backstop.
• explicit BECAUSE ← already handled by the connective backstop.
• SUPPORTS / CONTRADICTS ← local NLI (cross-encoder/nli-deberta-v3-xsmall, already loaded for Atropos): entailment→SUPPORTS, contradiction→CONTRADICTS. This is the genuinely new lever. Caveats: O(n²) → must cosine-gate; small-model contradiction recall is weak → high threshold, drop low-confidence to neutral.
• Only the residual implicit causal/logical edges (BECAUSE/IMPLIES with no explicit connective) escalate to Cerebras.
• Do NOT route typing to a small ≤7B LLM — they are ≈random on fine-grained relation typing (DeepSeek-R1-Distill-7B ≈0.34 acc); NLI+rules is the correct cheap path.
• Sources: LazyGraphRAG NLP-first microsoft.com/en-us/research/blog/lazygraphrag-setting-a-new-standard-for-quality-and-cost · NLI-for-edges + KG-embedding limits (inline in research notes).
• Impact: shrinks — often empties — the batched LLM call, leaving only true implicit causality for Cerebras.

Lever 3 — Defer the residual causal inference to an async background batch

Ack the write after extract + dedup + store + embed; run the (now-small) implicit-causal inference in a background batch — Lachesis already exists as a background stitcher, reuse it.

• Precedents: Mem0 add is async by design; GraphRAG builds the whole graph offline in batch (0 LLM at query); LazyGraphRAG defers all edge LLM work to query time.
• Guard the one real failure mode (a reader hitting an un-enriched neighbourhood) with a freshness flag + lazy on-read materialization of just that neighbourhood. Keep dedup + contradiction synchronous (correctness-critical).
• Sources: docs.mem0.ai/open-source/features/async-memory · microsoft.github.io/graphrag · Zep/Graphiti arxiv.org/html/2504.19413v1
• Impact: relation LLM cost leaves the write critical path entirely (→ write = extract + dedup).

Recommended shape to prototype & measure

1. Lever 1 first (batch infer) — biggest structural win, synchronous, contained blast radius.
2. Add Lever 2 (NLI + cosine + rules routing) — reserve Cerebras for implicit causality only.
3. Consider Lever 3 (defer) only if write-latency/cost is still a problem after 1+2, since it trades the "reader walks precomputed arrows" guarantee for a freshness window.

Net target: write-path LLM calls from 1 + 1 + N → ~2 flat (extract + dedup+batched-infer), or ~2 with relations fully deferred.

Success metrics (measure before committing)

• LLM calls per write: O(1), not O(N).
• $ / 1k atoms and write p50/p95 latency.
• Edge precision/recall vs current — the reasoning_extraction golden contract must not regress; add a fixture that counts inferred edges before/after.
• Read stays LLM-free; if Lever 3, quantify the enrichment-freshness window.

Non-goals / risks

• Not touching the read path.
• Not replacing extraction with LLM-free NLP (spaCy/FastGraphRAG) — extraction is ~75% of cost but quality-critical; keep 1 LLM extraction call.
• Batching can degrade with size/position bias → indices + strict schema + bounded batch.
• Async enrichment → eventual consistency → freshness flag + lazy materialization.

Research prompted 2026-07-05 after profiling showed infer_relations is O(N) per atom. Backed by 3 parallel web-research passes (batching / cheap-local-typing / async-defer) + FastThink synthesis.

[nikita-rulenko]

Lever 1 prototyped + measured (branch exp/96-batch-infer)

Batched infer_relations (N per-atom calls → 1 call over all atoms×candidates, chunked ≤12). Added a per-call token instrument (helixir::llm::cost). Measured a controlled seed + 3-fact follow write on cerebras gpt-oss-120b ($0.25/M in, $0.69/M out).

metric	BEFORE	AFTER	Δ
LLM calls	9	6	−33%
· extract	1	1	—
· decide	4 (1 batch +3 per-item fallback)	4	unchanged
· infer	4 (per atom)	1 (batched)	−75%
prompt tok	12,653	11,507	−9%
completion tok	5,525	5,443	−1.5%
total tok	18,178	16,950	−6.8%
$ / write	$0.00698	$0.00663	−4.9%

Quality: preserved. golden (diverse edge types), oracle (cross-write inferred edge), and reasoning_extraction guards all green on the batched path. Edge count 23→19 is run-variance (different candidate corpus per run), not a regression — the guards confirm the contract holds.

The bigger finding

The measurement showed the batch DECISION falls back to per-item for most atoms — here 3 of 4 atoms → 3 extra LLM calls (~6,700 prompt tokens), which is a larger cost than infer. Lever 1 does not touch it. Fixing decision-batch reliability (so gpt-oss returns all N decisions in one response) is the higher-value next lever: it would cut decide 4→1 and save ~6,700 prompt tokens/write, dwarfing the infer saving. Recommend a follow-up ticket.

Net: Lever 1 is a clean −33% call / −7% token / −5% $ win with preserved quality; the dominant remaining cost is the decision fallback.

[nikita-rulenko]

Lever 1.5 shipped (dev): decide_batch is now deterministic-reliable — dense local indices (the sparse-index renumbering trap is gone by construction) + ONE batched repair retry before any per-item fallback, raw response logged on parse failure. Units pin the contracts with a scripted provider (sparse gray → exactly 1 call; incomplete batch → exactly 2). Live: the Lever-1 measurement write now runs the optimal shape — 4 calls / $0.0038 (was 6 calls / $0.0066 with fallbacks, 9 / $0.0070 before Lever 1). Remaining in the umbrella: Lever 2 (route cheap edge types off the LLM: cosine/rules/NLI) and Lever 3 (defer residual causal inference to Lachesis async).

[nikita-rulenko]

Lever 2 shipped (dev): local NLI routes confident SUPPORTS/CONTRADICTS pairs off the LLM before the batched infer call. Two live-caught refinements baked in: bidirectional entailment (NLI is specific⊨general, our SUPPORTS isn't directional) and a shared-subject gate on contradictions (NLI is subject-blind — scored a cross-subject pair 0.99 contradiction; reuses the #93 shares_subject). Neutral/unconfident pairs still reach the LLM for implicit BECAUSE/IMPLIES. Self-gating on lean builds/missing model. Quality net fully green (cross-write oracle, reasoning_extraction 3/3, golden 4f). Cost effect is workload-dependent: restatement-heavy writes drop the infer call entirely; causal-heavy writes pay as before — by design, correctness over savings. Remaining: Lever 3 (defer residual causal inference to Lachesis async) — recommend evaluating AFTER real-world cost telemetry from Levers 1+1.5+2, per the memo's 'only if still needed'.