Replace JSON vector store with SQLite#6438
Conversation
This will add over a mb of binaries to the git history. Are we ok with that? Or did you mean you'll update the pr before it merges? |
|
I applied
* NO MERGE *
|
jeffhandley
left a comment
There was a problem hiding this comment.
This looks great to me, @MackinnonBuck. Nice work! Let's get a review from at least one of @roji or @SteveSandersonMS though, and wait for the NuGet packages to be available before merging.
roji
left a comment
There was a problem hiding this comment.
Thanks for pinging me, see below for a few comments.
Note that there will still be a few API changes for GA; some of my comments can already be addressed now, but others can't. So we can do another sync/iteration on this PR (hopefully by Friday everything will be in), and in general only merge after the GA nugets are fully available on nuget.org.
SteveSandersonMS
left a comment
There was a problem hiding this comment.
Looks superb - great job!
jeffhandley
left a comment
There was a problem hiding this comment.
Just the patch version bum in my comments, and then once @roji approves we're good to go. Nice work, @MackinnonBuck!
roji
left a comment
There was a problem hiding this comment.
Looks great @MackinnonBuck! See the one long comment below about doing automatic embedding generation for upsert as well, not just for search. Other than that I think it's ready for merging.
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
roji
left a comment
There was a problem hiding this comment.
Looks great @MackinnonBuck, well done!
Co-authored-by: Jeff Handley <jeffhandley@users.noreply.github.com>
This PR removes the JSON vector store and replaces it with a SQLite implementation using
Microsoft.SemanticKernel.Connectors.SqliteVec.Since
Microsoft.SemanticKernel.Connectors.SqliteVecisn't shipping publicly yet, I've builtMicrosoft.SemanticKernellocally and included the generated.nupkgs directly in the repo. This change (41bd61e) will get reverted after the updatedMicrosoft.SemanticKernelpackages ship.Microsoft Reviewers: Open in CodeFlow