Repository navigation
unstructured2graph: promote entity_type to real Memgraph labels, gated by an ontology - #248
Merged
Merged
Conversation
…d by an ontology LightRAG writes every extracted entity under one workspace label with entity_type as a plain property, so the graph never gets idiomatic Labeled-Property-Graph typing (:Person, :Organization, etc). Add an Ontology/EntityType vocabulary (DEFAULT_ONTOLOGY mirrors LightRAG's own built-in types) and promote_entity_types_to_labels(), wired automatically into ingestion: entities matching the ontology get an additional real label, additive only, never replacing the workspace label LightRAG's own upsert_node() relies on. Entities outside the ontology are left as-is rather than rejected, since deleting them would desync LightRAG's own KV/vector/ doc-status stores. The same Ontology can also steer LightRAG's extraction prompt itself via addon_params(), for callers who want a custom vocabulary. Also fixes MemgraphLightRAGWrapper.afinalize() to reset LightRAG's process-global shared-storage locks -- without it, a second real LightRAG instance created later in the same process reuses locks bound to a closed event loop and raises "bound to a different event loop" (surfaced by adding a second real-LLM e2e test).
…ce flagging Reworks the ontology-gating design from PR feedback: - Ontology now lives in a YAML config file, not hardcoded Python dataclasses. load_ontology(path) is the single source of truth for both LightRAG's extraction-steering (via addon_params()) and Memgraph's label-gating, so both stay coordinated by "same file path" rather than "hope two independently-built Ontology objects match". - from_texts()/from_unstructured() take ontology_path instead of a pre-built Ontology object, defaulting to a bundled default_ontology.yaml that mirrors LightRAG's own built-in type vocabulary. - promote_entity_types_to_labels() now stamps ontology_conformant=false on entities that don't match any type in the ontology, instead of leaving them silently indistinguishable from an unprocessed node. Conforming entities have the flag cleared, so re-running promotion after the ontology grows a new type corrects stale flags. Entities are never rejected -- the node and raw entity_type are always kept. Enforcement stays entirely application-level: Memgraph has no value-membership constraint, and its CREATE ENUM feature only validates writes that use an explicit cast syntax LightRAG never uses, so it isn't a viable backstop here.
There was a problem hiding this comment.
Pull request overview
This PR introduces an ontology-driven mechanism in unstructured2graph to promote LightRAG’s extracted entity_type property into real Memgraph labels (e.g. :Person, :Organization) while preserving the workspace label used for upserts. It also fixes a lifecycle bug in the lightrag-memgraph integration where process-global shared-storage locks weren’t reset during async finalization, and bumps package versions accordingly.
Changes:
- Add YAML-backed ontology loading (
load_ontology) plus a bundled default ontology to gate whichentity_typevalues are promoted to Memgraph labels. - Promote
entity_typeto labels during ingestion, and mark non-conforming entities withontology_conformant=false(and clear it when they later conform). - Fix
MemgraphLightRAGWrapper.afinalize()to reset LightRAG shared-storage state; bump related package versions/dependency floors.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| unstructured2graph/tests/test_ontology.py | Adds unit tests for ontology guidance, loading, defaults, and error cases. |
| unstructured2graph/tests/test_memgraph.py | Adds unit tests for label-promotion Cypher query generation and conformant flag handling. |
| unstructured2graph/tests/test_e2e_lightrag.py | Adds e2e coverage for default ontology label promotion and custom ontology nonconformance flagging. |
| unstructured2graph/src/unstructured2graph/ontology.py | Introduces ontology model + YAML loader + default ontology constant. |
| unstructured2graph/src/unstructured2graph/memgraph.py | Adds promote_entity_types_to_labels() implementation. |
| unstructured2graph/src/unstructured2graph/loaders.py | Wires ontology loading + label promotion into ingestion APIs (from_texts / from_unstructured). |
| unstructured2graph/src/unstructured2graph/default_ontology.yaml | Adds bundled default ontology vocabulary (mirroring LightRAG defaults). |
| unstructured2graph/src/unstructured2graph/init.py | Exports ontology APIs and bumps unstructured2graph version. |
| unstructured2graph/pyproject.toml | Bumps version and adds pyyaml dependency. |
| integrations/lightrag-memgraph/src/lightrag_memgraph/core.py | Fixes async finalization by resetting shared-storage state. |
| integrations/lightrag-memgraph/pyproject.toml | Bumps lightrag-memgraph version. |
| context-graph/sessions-graph/pyproject.toml | Bumps version and floors unstructured2graph>=0.5.0. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+65
to
+75
| if not isinstance(raw, dict) or "entity_types" not in raw: | ||
| raise ValueError(f"Ontology file {resolved_path} must be a YAML mapping with an 'entity_types' key") | ||
|
|
||
| entity_types = [] | ||
| for index, item in enumerate(raw["entity_types"]): | ||
| if not isinstance(item, dict) or "label" not in item or "description" not in item: | ||
| raise ValueError( | ||
| f"Ontology file {resolved_path}: entity_types[{index}] must be a mapping with 'label' and 'description'" | ||
| ) | ||
| entity_types.append(EntityType(label=item["label"], description=item["description"])) | ||
|
|
… review - Add enforce_ontology: bool = False to from_texts()/from_unstructured(). Ontology-based label promotion (and ontology_conformant flagging) now only runs when explicitly enabled; ontology_path is ignored (with a warning) if enforce_ontology isn't also set. Existing callers of from_texts()/from_unstructured() are unaffected until they opt in. Copilot review findings on PR #248, all addressed: - load_ontology(): restrict labels to safe Cypher identifier characters (they're f-string-interpolated into `SET n:{label}`), and validate entity_types is actually a list before iterating -- a null or mapping value previously raised a confusing TypeError instead of a clear ValueError. - promote_entity_types_to_labels(): guard the per-type SET with `AND NOT n:{label}` so re-running promotion doesn't rewrite nodes that already carry the label. - MemgraphLightRAGWrapper.afinalize(): reset shared-storage state in a finally block, so a transient error from finalize_storages() doesn't leave the process-global lock state stuck as initialized.
3 tasks
antejavor
added a commit
that referenced
this pull request
Jul 31, 2026
Closes #249. enforce_ontology previously conflated two concerns: promoting entity_type to a real Memgraph label, and restricting/validating that promotion against a fixed vocabulary. Splits them into two independent parameters: - promote_labels: bool = False -- promote every entity_type to a label (sanitized to a safe Cypher identifier, e.g. "natural object" -> :NaturalObject), no fixed vocabulary, no ontology_conformant flagging. - enforce_ontology: bool = False -- restrict promotion to ontology_path's vocabulary and flag anything outside it ontology_conformant=false. Takes precedence over promote_labels when both are set. New promote_all_entity_types_to_labels() in memgraph.py implements the unrestricted path: discovers distinct entity_type values under the workspace label (no static list to iterate, since there's no ontology), sanitizes each into a PascalCase label via _entity_type_to_label(), skipping values that can't be turned into a safe identifier. Wires sessions-graph's reconcile_session() to accept and forward the same three parameters, and updates its CLI (the real production entry point) to pass enforce_ontology=True, so reconciled session content actually gets typed labels going forward -- previously it called from_texts() with neither, so it got no promotion at all despite #248 landing. Note: unstructured2graph's real-LLM e2e suite (test_e2e_lightrag.py) and sessions-graph's test_e2e_reconciliation.py currently fail against a live Memgraph for an unrelated, pre-existing reason -- see #250 (MemgraphDocStatusStorage missing abstract methods required by lightrag-hku 1.5.5). Verified this reproduces identically on a clean main checkout before touching any code here. All non-e2e tests pass (60 in unstructured2graph, 24 in sessions-graph; 218 across the full 5-package suite).
This was referenced Jul 31, 2026
antejavor
added a commit
that referenced
this pull request
Aug 4, 2026
) 1.5.5 (published 2026-07-31, same day as PR #248) added new @AbstractMethod requirements to DocStatusStorage as part of a "memory- bounded scheduling" overhaul upstream (get_docs_by_statuses_page, get_docs_by_ids, get_full_docs_by_ids, resolve_doc_source_strict). MemgraphDocStatusStorage doesn't implement them yet, so constructing a real LightRAG instance raises TypeError -- breaking every real-LLM e2e test in unstructured2graph and sessions-graph, in CI and locally (#250). Pins lightrag-hku==1.5.4 (previously an open >=1.5,<1.6 floor) until the new interface is implemented. Also updates uv.lock, which had drifted noticeably out of sync with pyproject.toml's already-committed versions (lightrag-memgraph, mcp-memgraph, memgraph-toolbox, sessions-graph, unstructured2graph all resolved to stale pins before this). Verified: all real-LLM e2e tests pass again (unstructured2graph's test_e2e_lightrag.py, sessions-graph's test_e2e_reconciliation.py), full 5-package suite green (218 tests). #250 stays open to track the real fix: implementing the new DocStatusStorage interface so lightrag-hku can be unpinned.
antejavor
added a commit
that referenced
this pull request
Aug 4, 2026
Closes #249. enforce_ontology previously conflated two concerns: promoting entity_type to a real Memgraph label, and restricting/validating that promotion against a fixed vocabulary. Splits them into two independent parameters: - promote_labels: bool = False -- promote every entity_type to a label (sanitized to a safe Cypher identifier, e.g. "natural object" -> :NaturalObject), no fixed vocabulary, no ontology_conformant flagging. - enforce_ontology: bool = False -- restrict promotion to ontology_path's vocabulary and flag anything outside it ontology_conformant=false. Takes precedence over promote_labels when both are set. New promote_all_entity_types_to_labels() in memgraph.py implements the unrestricted path: discovers distinct entity_type values under the workspace label (no static list to iterate, since there's no ontology), sanitizes each into a PascalCase label via _entity_type_to_label(), skipping values that can't be turned into a safe identifier. Wires sessions-graph's reconcile_session() to accept and forward the same three parameters, and updates its CLI (the real production entry point) to pass enforce_ontology=True, so reconciled session content actually gets typed labels going forward -- previously it called from_texts() with neither, so it got no promotion at all despite #248 landing. Note: unstructured2graph's real-LLM e2e suite (test_e2e_lightrag.py) and sessions-graph's test_e2e_reconciliation.py currently fail against a live Memgraph for an unrelated, pre-existing reason -- see #250 (MemgraphDocStatusStorage missing abstract methods required by lightrag-hku 1.5.5). Verified this reproduces identically on a clean main checkout before touching any code here. All non-e2e tests pass (60 in unstructured2graph, 24 in sessions-graph; 218 across the full 5-package suite).
Merged
3 tasks done
antejavor
added a commit
that referenced
this pull request
Aug 18, 2026
…try_points registry) (#295) PyPI has been stale for months relative to main -- these four packages haven't been released since before most of maps #275/#288's work landed, so the actual subagent-nesting model (Agent node, SPAWNED inference, HAS_AGENT, USED_SKILL-on-Agent attachment) isn't published anywhere yet. - actions-graph 0.1.2 -> 0.2.0: first-class Agent node model, SPAWNED inference rule, and the agent_spawning_tool_names default-name fix, all verified live against a real Claude Code session. - skills-graph 0.1.3 -> 0.2.0: USED_SKILL attaching to the specific Agent (not just the flat Session) when skill usage happens inside a subagent. - sessions-graph 0.4.0 -> 0.5.0: reconcile_session() now writes episodic Session.summary, plus config propagation fixes to the reconcile subprocess. (0.4.0 itself was already unreleased -- the ontology/label- promotion work from #248/#251.) - agent-context-graph 0.1.9 -> 0.2.0: entry_points-based runtime plugin registry. (main's 0.1.9 already diverged from the last published 0.1.9 build without a version bump -- #273 landed after the last release.) No code changes -- version bumps only, verified compatible with existing consumers (all internal >= constraints, no upper bounds).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LightRAG writes every extracted entity under a single workspace label (e.g.
:base), withentity_typeonly as a flat property — so the graph never gets idiomatic Labeled-Property-Graph typing (:Person,:Organization,:Location, ...). This adds ontology-gated promotion ofentity_typeto a real Memgraph label, opt-in as part of ingestion:enforce_ontology: bool = False. Off by default — entities are left exactly as LightRAG wrote them, no label promotion, noontology_conformantflagging, no behavior change for existing callers. WhenTrue, promotion runs usingontology_path(or a bundled default if omitted).load_ontology(path)parses a YAML file (entity_types: [{label, description}, ...]) into anOntology. Both LightRAG's extraction-steering (ontology.addon_params(), wired intoMemgraphLightRAGWrapper.initialize()) and Memgraph's label-gating load the same file independently, so they stay coordinated by "same file path" rather than by passing a shared in-memory object between two different call sites.promote_entity_types_to_labels(): additive only —SET n:{Label}for entities whoseentity_typematches the ontology (skipping nodes that already carry the label), never touching the workspace label LightRAG's ownupsert_node()relies on to re-MERGEfuture updates.entity_typeare always kept, stampedontology_conformant: falseinstead of being silently indistinguishable from an unprocessed node. Re-running promotion (e.g. after the ontology grows a new type) clears the flag on anything that now conforms.ASSERT n.prop IN [...]isn't valid Cypher), andCREATE ENUM— while real and genuinely rejecting at write time — only engages for writes using an explicitEnumName::Valuecast, which LightRAG never does, plus it's additive-only in this Memgraph version (REMOVE VALUE/DROP ENUMunimplemented).CONTEXT.md): entity resolution (LightRAG's own merge is exact-name-match and destructively collapses conflictingentity_typevalues), a stable-core/volatile-extension ontology hierarchy, and standalone re-scan/retraction tooling for renamed or merged types.Also fixes two real bugs found while building this:
MemgraphLightRAGWrapper.afinalize()never reset LightRAG's process-global shared-storage locks, so a second realLightRAGinstance in the same process reused locks bound to a closed event loop (RuntimeError: ... bound to a different event loop). Fixed by callingfinalize_share_data()in afinallyblock, so it resets even iffinalize_storages()itself raises (per GitHub Copilot's review).load_ontology()didn't validate that alabelis a safe Cypher identifier before f-string-interpolating it intoSET n:{label}— a malformed or malicious label could break or inject into the generated query. Now validated up front with a clearValueError(per GitHub Copilot's review, along with a fix forentity_types: null/non-list values raising a confusingTypeErrorinstead of a clear error).Version bumps:
unstructured2graph0.4.0 → 0.5.0,lightrag-memgraph0.3.0 → 0.3.1,sessions-graph0.2.0 → 0.3.0 (floor bumped tounstructured2graph>=0.5.0— note sessions-graph's own reconciliation call doesn't passenforce_ontology=Trueyet, so it won't get label promotion until that's wired up separately).Test plan
unstructured2graphunit tests — ontology YAML loading/validation (including the new label-safety and shape checks), label promotion (including the already-labeled skip guard),ontology_conformantflagging/clearing,enforce_ontologydefault-off/opt-in behavior.unstructured2graphe2e tests against live Memgraph + real OpenAI calls — no promotion by default, default-ontology promotion when enforced, and a narrow custom-ontology file correctly flagging out-of-vocabulary entities.lightrag-memgraphunit tests —afinalize()resets shared state both on success and whenfinalize_storages()raises.default_ontology.yamlships inside the built wheel.scripts/dev-memgraph.sh test) across all five packages — 49 passed.Not yet merged — still open for further review/polish.