feat: add document structure into GraphRAG #2033

KingSkyLi · 2024-09-20T08:02:36Z

Aries-ckt · 2024-09-23T10:59:24Z

it looks like db upsert error.

 document embedding, failed:xx.txt, Query execution failed: {code: ParserException} {message: ParserException: line 3:12 no

 viable alternative at input 'CALL db.upsertVertex("chunk", [{id: "62c3475f-3ecf-4520-ae97-88336aaca5fe", name: 

"none_header_chunk", content: d748118572'}

dbgpt/storage/knowledge_graph/community_summary.py

dbgpt/storage/graph_store/graph.py

dbgpt/storage/knowledge_graph/community_summary.py

dbgpt/storage/knowledge_graph/knowledge_graph.py

dbgpt/rag/transformer/graph_extractor.py

dbgpt/storage/graph_store/tugraph_store.py

dbgpt/storage/knowledge_graph/community_summary.py

dbgpt/storage/knowledge_graph/base.py

Appointat · 2024-09-29T01:53:24Z

@KingSkyLi Hi, I was assigned to help review this pr, could you please explain what kind of the doc structure needed to be added in the graph? Thanks a lot!

…uGraphStore

…limit parameters

…e code

Aries-ckt · 2024-10-17T10:08:36Z

@KingSkyLi @Appointat
i have met the problem when upload file.

2024-10-17 18:06:45 B-V0ECMD6R-0244.local dbgpt.serve.rag.service.service[90267] ERROR document embedding, failed:读写分离.md, 'Vertex' object has no attribute 'type'

fanzhidongyzby

Very good refactoring work, although there are still some further improvement points.

dbgpt/storage/graph_store/graph.py

dbgpt/storage/graph_store/tugraph_store.py

dbgpt/storage/knowledge_graph/community_summary.py

Aries-ckt · 2024-10-17T14:46:47Z

Test Success

Aries-ckt

LGTM.

Appointat · 2024-10-18T02:11:07Z

This reverts commit 9693984.

Co-authored-by: tpoisonooo <khj.application@aliyun.com> Co-authored-by: vritser <vritser@163.com>

csunny

LGTM

Aries-ckt

LGTM

fanzhidongyzby

fix comments below by another pr

fanzhidongyzby · 2024-10-18T10:14:55Z

dbgpt/datasource/conn_tugraph.py


    def delete_graph(self, graph_name: str) -> None:
-        """Delete a graph."""
+        """Delete a graph in the Neo4j database if it exists."""


wrong comment

fanzhidongyzby · 2024-10-18T10:17:38Z

dbgpt/storage/graph_store/graph.py

+
+    DOCUMENT = "document"
+    CHUNK = "chunk"
+    ENTITY = "entity"  # view as general vertex in the general case


should be described as default node type in knowledge graph

fanzhidongyzby · 2024-10-18T10:18:22Z

dbgpt/storage/graph_store/graph.py

+    DOCUMENT = "document"
+    CHUNK = "chunk"
+    ENTITY = "entity"  # view as general vertex in the general case
+    RELATION = "relation"  # view as general edge in the general case


should be described as default edge type in knowledge graph

fanzhidongyzby · 2024-10-18T10:19:32Z

dbgpt/storage/graph_store/graph.py

+
+    def is_edge(self) -> bool:
+        """Check if the element is an edge."""
+        return not self.is_vertex()


not vertex != edge, enumerate all valid edge types here

fanzhidongyzby · 2024-10-18T10:20:56Z

dbgpt/storage/knowledge_graph/community/memgraph_store_adapter.py

+
+    async def discover_communities(self, **kwargs) -> List[str]:
+        """Run community discovery with leiden."""
+        pass


fanzhidongyzby · 2024-10-18T10:41:59Z

dbgpt/storage/knowledge_graph/community_summary.py

+                )
+
+                for graph in graphs:
+                    graph_of_all.upsert_graph(graph)


update graph edge _chunk_id

fanzhidongyzby · 2024-10-18T10:42:19Z

dbgpt/storage/knowledge_graph/community_summary.py

+
+        self._graph_store_apdater.upsert_graph(graph_of_all)
+
+        # use asyncio.gather


remove unused comments

fanzhidongyzby · 2024-10-18T10:46:27Z

dbgpt/storage/knowledge_graph/community_summary.py

-            for graph in graphs:
-                self._graph_store.insert_graph(graph)
+        # Support graph search by the document and the chunks
+        if self._graph_store.get_config().enable_document_graph:


move this part of code into _parse_chunks, and rename it to load_document_graph(chunks) -> List[Chunk]

fanzhidongyzby · 2024-10-18T11:06:14Z

dbgpt/storage/knowledge_graph/community_summary.py

+
+                subgraph_for_doc = self._graph_store_apdater.explore(
+                    subs=keywords_for_document_graph,
+                    limit=5,


use config: KNOWLEDGE_GRAPH_CHUNK_SEARCH_TOP_SIZE

fanzhidongyzby · 2024-10-18T11:09:39Z

dbgpt/storage/knowledge_graph/community/tugraph_store_adapter.py

+        if not subs:
+            return MemoryGraph()
+
+        if depth is None or depth < 0 or depth > self.MAX_HIERARCHY_LEVEL:


depth = 3 by default

KingSkyLi added 2 commits September 14, 2024 16:24

Graph RAG add document and chunk type

0bb7596

fixed insert_graph

961f7ab

KingSkyLi added 3 commits September 25, 2024 11:25

get source file name

bc5ffda

add delete document;

d171735

change create label function, graph_vis api, local query

ecbc3fd

fanzhidongyzby linked an issue Sep 27, 2024 that may be closed by this pull request

[Feature][GraphRAG] Knowledge graph extraction supports document structure information #2007

Closed

2 tasks

fanzhidongyzby changed the title ~~Add the document structure of the graph.~~ feat: add document structure into GraphRAG Sep 27, 2024

github-actions bot added the enhancement New feature or request label Sep 27, 2024

fanzhidongyzby added the GraphRAG Module: GraphRAG label Sep 27, 2024

fanzhidongyzby reviewed Sep 27, 2024

View reviewed changes

add next edge;

e3f9c09

Appointat reviewed Sep 28, 2024

View reviewed changes

dbgpt/storage/knowledge_graph/base.py Outdated Show resolved Hide resolved

Appointat added 7 commits October 11, 2024 11:47

refactor: Refactor graph_vis API and local query functions

80c69a2

Refactor graph store initialization and label creation

c368524

refactor: refactor graph store initialization and label creation

7dab7ed

refactor: Refactor graph store initialization and label creation

442a5f9

refactor: Refactor graph store initialization and label creation

fad9e0b

refactor: Refactor graph store initialization and label creation

f0d970c

Refactor document vertex type description in TuGraphStoreConfig

e8cb3cb

fanzhidongyzby added the hacktoberfest label Oct 11, 2024

Appointat added 8 commits October 12, 2024 00:06

refactor: the great refactor

9b560f2

refacor: Refactor graph store factory and TuGraphStore

53277ff

Refactor property filtering in TuGraphStore

ee4cfc0

refactor: Refactor graph store initialization and label creation in T…

733f1d5

…uGraphStore

refactor: Refactor graph store tests and adapters

8d3306b

Refactor import statements in graph_store/base.py

68c20ce

Refactor graph store adapter in CommunityStore

072ca7f

chore: Refactor import statements in graph_store/base.py

1f2fb6f

Appointat added 2 commits October 17, 2024 00:34

fix: Refactor TuGraphStoreAdapter explore method to handle depth and …

7feea09

…limit parameters

feat: enable graph search for documents and entities, and refactor th…

1dd6683

…e code

Appointat mentioned this pull request Oct 17, 2024

fix: tugraph create schema error #2073

Closed

6 tasks

Appointat added 2 commits October 17, 2024 16:52

feat: Refactor explore_text_link method in GraphStoreAdapter

4b92714

Refactor explore_text_link method in GraphStoreAdapter

91962d0

fanzhidongyzby reviewed Oct 17, 2024

View reviewed changes

fix: fix mypy

5ba1215

Appointat added 2 commits October 17, 2024 22:49

fix

3ccfc3d

fix by fmt

1c9edcd

Aries-ckt previously approved these changes Oct 18, 2024

View reviewed changes

fix: fix bugs

869556f

Appointat dismissed Aries-ckt’s stale review via 869556f October 18, 2024 06:30

Appointat added 3 commits October 18, 2024 14:33

fix by black format

9693984

Revert "fix by black format"

cc588de

This reverts commit 9693984.

fix by make fmt

59aab63

Appointat mentioned this pull request Oct 18, 2024

fix(tugraph_store.py): _create_schema crash #2079

Closed

6 tasks

Appointat and others added 5 commits October 18, 2024 15:19

fix

8f7089e

chore: fix and update the doc

d153132

doc

5ab6345

Co-authored-by: tpoisonooo <khj.application@aliyun.com> Co-authored-by: vritser <vritser@163.com>

doc: add the image

847c4c6

fix

943db44

csunny approved these changes Oct 18, 2024

View reviewed changes

Aries-ckt approved these changes Oct 18, 2024

View reviewed changes

Aries-ckt merged commit 88e3d12 into eosphoros-ai:main Oct 18, 2024
4 checks passed

Aries-ckt added the hacktoberfest-accepted hacktoberfest-accepted label Oct 18, 2024

fanzhidongyzby reviewed Oct 18, 2024

View reviewed changes

Appointat deleted the feature-textlink branch November 12, 2024 03:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add document structure into GraphRAG #2033

feat: add document structure into GraphRAG #2033

KingSkyLi commented Sep 20, 2024

Aries-ckt commented Sep 23, 2024

Appointat commented Sep 29, 2024

Aries-ckt commented Oct 17, 2024

fanzhidongyzby left a comment

Aries-ckt commented Oct 17, 2024

Aries-ckt left a comment

Appointat commented Oct 18, 2024

csunny left a comment

Aries-ckt left a comment

fanzhidongyzby left a comment

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024

fanzhidongyzby Oct 18, 2024

Appointat Oct 20, 2024


		self._graph_store_apdater.upsert_graph(graph_of_all)

		# use asyncio.gather

feat: add document structure into GraphRAG #2033

feat: add document structure into GraphRAG #2033

Conversation

KingSkyLi commented Sep 20, 2024

Aries-ckt commented Sep 23, 2024

Appointat commented Sep 29, 2024

Aries-ckt commented Oct 17, 2024

fanzhidongyzby left a comment

Choose a reason for hiding this comment

Aries-ckt commented Oct 17, 2024

Aries-ckt left a comment

Choose a reason for hiding this comment

Appointat commented Oct 18, 2024

csunny left a comment

Choose a reason for hiding this comment

Aries-ckt left a comment

Choose a reason for hiding this comment

fanzhidongyzby left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment