Provide more details on embedding

[jayrajput]

OpenAI embedding looks like a neat idea. I gave it a try. But I am not sure if it effective for my notes. Where can one learn more about Open AI embedding and its usage in ekg. I have not yet read the complete Ekg manual and I wonder if some details are already provided in their (plan to dedicate time on that tomorrow_.

It will be good to get some example usage documented or explained using videos to show the power of this feature. May be Andrew can do a video showing embedding with his notes or something on these lines.

Other reason I want to learn more is to make sure that there is no Ekg bug with my notes database hindering me to utilize the power of Open AI embedding.

[ahyatt]

I did talk about this and demo it out a bit in one of my previous videos (see the part here). I do explain it a bit in the ekg manual, but it was more of a technical explanation, and I agree that a more user-focused explanation would be good.

I'll write a bit more in the manual, and perhaps a more focused video would be a good idea as well, thank you for the suggestions.

[ahyatt]

I've checked in 9dee807, which focuses the text in the manual more on what embeddings are used for and why they are useful. Hopefully that's a start, but further feedback will be welcome.

[jayrajput]

Thanks. I read through the changes in the manual. The example seems apt and the documentation make sense to me. I was not aware about LLP (consider me living under the rock). I went through the video and understand it better now. Itching to try the capabilities, but I broke my home laptop emacs configuration (the sqlite dependency one), so it will take some time before I fix it and start using it. May be I will just move to emacs29 (seems like there is a windows pretest available) and see how it goes. Thanks again!!

[jayrajput]

I uninstalled emacs 28 and installed emacs 29 pretest and things worked out of the box. I gave ekg-embedding-search a try and read more about openai embedding online. I now have a better understanding and quirkines of the embedding search. For example, ekg-embedding-search will not help with text search. For some reason, I thought it can search through the text, which it does not.

• Search for carpenter will not find notes with my carpenter details.
• Search for health was able to get my list of quarterly blood test which I go through.

Thanks for the awesome feature.

[jayrajput]

Reading more on the embeddings, I wonder if clustering make more sense to the ekg use case. It will be similar to org-roam-ui to see a cluster of semantically related notes together.

Further I think the ekg software is doing embedding just on the text. Will it be nice to do embeddings on the tag text itself?

• @ahyatt

Reference: https://platform.openai.com/docs/guides/embeddings/use-cases

[jayrajput]

Further the notes are not chunked and I am not sure if that make a difference in generating embeddings and doing the comparison for search later.

[jayrajput]

Reading more on embedding, I believe that any metadata from the notes shall also be removed. For examples, my notes exported from org-roam has these metadata at the top which shall be removed to make the embeddings more effective.

:PROPERTIES:
:ID:       744cc648-ae83-4eb5-998e-b8972a32dbc2
:END: