Vector Embedding Brainstorming

[Shubham-Khichi]

You all know that I'm not only building DevDocs for developers but also for folks who want to get into LLM tech. One of the primary goals of our company products is Simplicity Of Use: "How can we build a product that even a grandma can use it" kind of simplicity.

Every decision made is to make DevDocs and our other products easy to use, every feature is designed and refined to ensure that after few iteration this feature becomes super intuitive.

Vector databases which is our 3rd most requested feature is conflicting with our notion of simplicity of use because the inherent complexity of chunking, embedding and setting up a vector db on top of configuring so many parameters makes this process difficult, not to mention giving these parameters in the hands of folks to play around and sometimes mess up the variables and now your data doesn't get properly pulled by the LLM.

So traditional vector storage methods will fail us. So I am leaning towards using Copali to bypass the complex structure of websites, context management and making sure that embeddings are happening right.

Here is the thought process:

1. We will convert the website into PDF (md and json formats will still be available if you want to create datasets to finetune)

2. A user will input their vector storage APIs into the UI which will remain persistent during their entire account history.

3. Provide a URL and let DevDocs ensure it spiders, scrapes, loads into MCP and create Md, json and PDF formats

4. With the PDF version DevDocs uses a local vision model to use Copali's capabilities to embed the entire page of technical documentation into the vector storage so that CONTEXT is not lost during retrieval.

5. Victory

Reason for this approach is technical documents are already structured in a page. Meaning that 1 concept is explained in 1 page, if we embed the whole page then the concept and its relevant examples are not lost.

What are your thoughts on this. Suggestions to enhance or better solution is always welcomed. Ok

[TomLucidor]

PDF is a waste of space, and harsh on parsing. Unless it is for documentation of polished products, PDFs are generally a bad idea. JSON, YAML, MarkDown etc. all have tradeoffs but generally optimistic on lightweight markup

[Shubham-Khichi]

I agree with you on PDFs since they were made for humans not machines. Since a lot of human data is stored in PDFs like books, contents, etc there should be a feature to digest it. Luckily OCR and embedding technology has improved a lot to parse the data from PDFs.

[TomLucidor]

If you are talking about raw data ingestion/extraction, this makes more sense. Don't make PDF just to re-chew

[Shubham-Khichi]

If you are talking about raw data ingestion/extraction, this makes more sense.

Yes raw data extraction and storage