Edges dropped when source endpoint is missing from the current batch's entity list, despite the entity existing earlier — likely batch-local resolution instead of global entity lookup

[totalslacker]

Symptom

During an ingest, the console log fills with lines like:

liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Creator Studio' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'iMessage' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'FaceTime' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'AirTag 2' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'M5 MacBook Pro' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'iOS 26' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Civilization VII' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Retrocade' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Google Gemini' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Apple AirDrop' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Apple N1 Wi-Fi chip' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Vision Pro M5' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Mac Pro' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'Apple CarPlay' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'iPadOS 26' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'iPad Pro' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'macOS 26.1' (src_in_list=false, dst_in_list=true)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Apple' → 'ICEBlock' (src_in_list=false, dst_in_list=true)

In every case the destination endpoint IS present in the batch's entity list (dst_in_list=true) but the source is not (src_in_list=false). The source is 'Apple' in every one of these examples — a very common, high-recurrence entity that would clearly have been created in earlier ingest batches from other documents.

Why this looks like a bug rather than expected behavior

• 'Apple' is a canonical entity that gets created on almost any tech-related ingest. It is overwhelmingly likely to already exist in the graph by the time these later edges are being resolved.
• The src_in_list=false field name suggests endpoint resolution is scoped to a per-batch entity list (the entities being written in this ingest call), rather than a global entity-table lookup. If it were the latter, previously-created 'Apple' would resolve fine.
• The pattern is consistent: it's always cross-references TO an existing entity that get dropped, never cross-references FROM one — because the local batch has just extracted the destination entities (they are in the current list) but references an earlier-extracted source entity that is not in the current list.

Hypothesis

The edge-resolution step during ingest checks a local (batch-scoped) entity list for both endpoints. When the source endpoint was created in a previous batch, its ID isn't in the current batch's list, so resolution fails and the edge is dropped — even though a lookup against the global entity table would find it.

If this hypothesis is right, the fix is to fall back to a global entity lookup (by name, or by whatever canonical key) when the batch-local lookup misses. Any missing endpoint that exists globally should be resolved to its existing ID rather than dropped.

Impact

For a corpus with recurring canonical entities (Apple, Google, Microsoft, common frameworks, etc.), a significant fraction of graph edges silently disappear. The graph ends up with correct nodes but a sparse and inconsistent edge set — cross-document connections through popular hubs are systematically missing.

Downstream consequences: any knowledge_get_entity_neighbors-style query underreports connectivity, "related concepts" walks produce shallower results, and the graph's usefulness as a cross-source semantic map is degraded exactly at the points that would otherwise be most connected.

Suggested investigation

1. Grep for the log message string dropping edge with unresolvable endpoint — that's the emission site.
2. Trace back to where src_in_list / dst_in_list are computed. That's the resolution logic.
3. Determine whether it consults the entity table at all, or only the batch-local list.
4. If only local: add a global fallback lookup for the "unresolved but might exist globally" case.

Environment

• liminis-context-graph running from ~/.lcg/ (.lcg/service.sock, .lcg/db/, .lcg/wal/).
• Called via WBPP through the WebBrain plugin adapter (which translates wb_ingest_source → knowledge_process_chunk).
• Ingests happening in bursts during WebBrain's browsing-triggered indexing pipeline. Batches are chunk-sized (one WBPP call per source page).
• Observed after nuking the liminis DB and re-syncing WebBrain's corpus from scratch, so this reproduces on a fresh graph without any prior state.

Related upstream context

This was surfaced while integrating liminis with WebBrain's plugin bus. WebBrain's side of the diagnosis lives at github.com/totalslacker/WebBrain — happy to share more context or a repro walkthrough if useful.

[verveguy]

Thanks @totalslacker — your hypothesis is exactly right. Confirmed: edge-endpoint resolution is batch-local (episode.rs:190-216 builds the name set only from the current chunk's extracted entities; the global get_entity_by_name_ci resolution happens later, in Phase B, after the edge is already dropped).

One thing your diagnosis didn't have: there are two silent-drop sites — the validation drop and Phase C's name_to_uuid map (episode.rs:432-449), which also rebuilds only from the batch. The fix (group-scoped global fallback) has to cover both.

Work issue #209 tracks the fix (blocked on #208 below, since both edit episode.rs).

[verveguy]

Fixed and merged to main (#209 → PR #218, 04aacec).

Both drop sites are addressed — the validation predicate and a second silent drop at commit time in Phase C's endpoint→uuid map that your report's diagnosis hadn't surfaced. Resolution now falls back to the persisted Entity table, scoped to group_id so same-named entities in different groups never cross-link. Tests cover the cross-batch case, group isolation, and the still-unresolvable case.

One thing your log grep will notice: the message now reads src_resolved=/dst_resolved= instead of src_in_list=/dst_in_list=, since "in list" is no longer the whole story.

This is in main but not yet in a release — we'll follow up here when it ships in a tagged version. Thanks again for the sharp diagnosis.

[totalslacker]

Still getting dropped edges with PR #218 so there must be another case for this:
liminis-context-graph: dropping edge with unresolvable endpoint: 'Kigali Amendment' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'James Hansen' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'School Strike for Climate' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Extinction Rebellion' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'World Health Organization' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'World Bank' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'carbon capture and storage' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'solar radiation modification' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Atlantic meridional overturning circulation' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'Amazon rainforest' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'permafrost thawing' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: dropping edge with unresolvable endpoint: 'ocean acidification' → 'climate change' (src_resolved=true, dst_resolved=false)
liminis-context-graph: entity resolution: type conflict for 'Atlantic meridional overturning circulation': existing labels ["Entity", "System"], extracted type 'Process'