Please add your ideas - infrastructure planing thread

[nilsherzig]

Please comment if you have any thoughts on this:

We have a "chat layer" which has a history of the user's prompts and messages (green and purple). If needed, this chat chain can call other chains, which run autonomous without user interaction by using a self critique loop.

It would be very easy to add more chains to this, like a "programming chain" using deepseek-coder as the LLM.

I would have to come up with a config format for these chains, but essentially they are just a couple of conditions and strings in / strings out.

[romanr]

Aren’t GitHub Discussions better suited for this?

[nilsherzig]

good point, there you go

[priyanmuthu]

I like the idea of tools. I'm very interested in the programming chain, but the challenge is to run it in a sandboxed environment I suppose?

[nilsherzig]

yes we would need a solution for this, but there are sandboxing solutions out there and interpreted languages like python can be "filtered".

[andrejguran]

Love the project and idea to open source perplexity 👍
Can you describe a little how does the vector DB get populated with that, what data and how the results are being processed?

[nilsherzig]

I think i will write a more detailed post on this. But essentially:

1. downloads top n websites
2. removed html tags
3. removed some other unwanted things
4. splits resulting text into little chunks (tries to split at new lines or new paragraphs) + chunks have some overlap
5. chunks are saved into the vector db
6. vector db returns top n matches on query

[arronKler]

if the search process is a one round task, why not just remove vector db component and just do text embedding and similarity search in the working memory.
Or maybe just using a ColBert model here is enough to get most relevant text.

[tslmy]

Is this "retrieval augmented generation" (RAG), and -- with tools -- "LLM agent"? I didn't see these terms mentioned in the document, so I was wondering if I'm missing something.

[nilsherzig]

Assuming my understanding of these terms is right, yes.

I have no real education on the whole LLM topic so I avoided the use of specific terminology :)

[Viibrant]

Yes it sounds like it, I thought I missed something as well

[aagha]

I'd love the ability to add local folders (Obsidian notes), Google Drive/Dropbox locations to search.

[nilsherzig]

Yes, i think that's a great idea and will probably be the next tool :)

I got it working with just markdown (since im also using obsidian). But im still working on adding csv and pdf support.

[twilwa]

seconding this -- additionally, llava1.6 works in ollama, so you can vectorize pdfs, screenshots, screen recordings, etc.

[nilsherzig]

oh thats very good to know, thanks :)

[Zackaryia]

Also being able to scan my entire disk (terrabytes+) and allowing me to find files ive forgotten or search in vector space for my data would be cool.

I would like a more traditional non-llm 10 blue links type of search through the vector db though. I find that particularly useful for finding a photo or video or document that is on my disk but I just cant find it, or i want to agregate all files of a certain type.

Although some data would be more important like my obsidian vault, vs random files on disk so a weighting system may be needed but in practice that might not be true.

[zaggynl]

I combined this with distrobox to setup an ubuntu linux container for rocmn support with 7900xtx in ollama:

Used the ini from this ROCm/ROCm#2990 (comment)
distrobox assemble create --file distrobox.ini
distrobox enter rocm-rl
get the ollama install.sh file
modify the ollama install file to include
Environment="OLLAMA_HOST=0.0.0.0
install it

It does need some limits, this froze up my machine :D
"Action: Calculator. Action Input: list(range(999999999))"

[nilsherzig]

Hahaha thanks for reporting. The calculator tool isn't mine (just to shift some blame hehe) but I'm going to add some sort of limiter.

In the meantime you can specify how much resources a docker container can use (it's just some extra lines inside the compose file).

[xx77yy]

what does it uses to search on internet?

[Technetium1]

https://github.com/searxng/searxng. You can see some random instances available for testing it here: https://searx.space/

[Zackaryia]

Non web data sources.

• Wikidata
• Wikipedia
• Wolframalpha
• Khanacademy
• Specific library documentation
• Youtube video captions
• Wolfram Data Sources https://datarepository.wolframcloud.com
• Local Data

Just a lot of places that you can pull data from. Maybe creating a data template / format that can be ingested into the vector db would be ideal, then writing downloadeders that take the data and reformat it for the vector DB would be the easiest way to implement this.

Also I would like to see an open source replacement to Wolfram Alpha, specifically the math solving part of it but that seems like a different much harder task unto itself.

[MakkiLoyola]

Hey, im going to recreate the entire Front End of Perplexity, ill post it here once done,

Should be done by the 26th of April

[chymian]

I open discussion on partitioning the memory/vectordb:

• to keep an overview, make topic-specivic partitions, like a workspaces
• long term memory (like curated stuff, i.e. obsidian)
• clearable short term, like failed research attempts, which would spoil further querys
• chat-history

and thankyou @nilsherzig this is working very very well, even in that early state.

[chymian]

add an url for embedding models.

to overcome the need to load/unload/change model permanently, introduce a embedding-url

i.e. we can run a

• second ollama on that endpoint.
• TEI huggingface TextEmbeding Interface.
• ...

text embeddings are needed by many AI-APPS, so that it's usefull to have a dedicated endpoint on your Machine.
and since the models are small, one can keep them load (on GPU) or running on CPU is ok. (here ;)

[PriNova]

For those interested but didn't know that it exists.
Web scraping for LLM compatible reading use jina reader:
For any open public websites, simply prepend http://r.jina.ai as URL

It is totally free
As example of the original website HNSW Wikipedia
And here with the jina reader: Jina HNSW Wikipedia Format

So all beatifulsoaping is obsolete even images will be converted with ALT text.