feat: transforms for Knowledge Graphs #1345
Conversation
dosubot
bot
added
the
size:XL
|
|
||
|
|
||
| # This regex pattern matches URLs, including those starting with "http://", "https://", or "www." | ||
| links_extractor_pattern = r"(?i)\b(?:https?://|www\.)\S+\b" |
There was a problem hiding this comment.
here we are initing these extractors - should we change that?
There was a problem hiding this comment.
Yes, please. Such inits can't be scaled cleanly.
|
|
||
|
|
||
| @dataclass | ||
| class HeadlineSplitter(Splitter): |
There was a problem hiding this comment.
This I see handles single-level headline splitting, ideally, we might need an iterative one but let's leave that for now. Also, we would need normal hierarchical splitters like ones in llama-index or langchain. This is because headline extraction from documents such as PDS is error prone, so we can't use headline splitters with large pdf files prly. [This can be added later]
There was a problem hiding this comment.
I've noted these down these will have to address them is separate PRs
| headlines: t.Dict[str, t.List[str]] | ||
|
|
||
|
|
||
| class HeadlinesExtractorPrompt(PydanticPrompt[StringIO, Headlines]): |
There was a problem hiding this comment.
Can you replace the example here with a more generalizable and shorter one? this one targets Arxiv documents specifically.
Transforms
There are basically 3 type of transforms you can do
ExtratorsSplittersRelationshipBuilder