Common/unified EF experience for vector search

[roji]

EF does not currently have a unified vector search story: vector search is implemented separately for each provider, exposing provider-specific search functions. While this allows exposing each database's exact support, there may be value in a common/unified experience, much like how we try to capture other common database functionality in relational and core. At this point most databases have some sort of vector search story (it's practically universal), and we have enough examples to try to triangulate a unified thing. Microsoft.Extension.VectorData (MEVD) also showed that it's possible to successfully abstract over the different databases, and EF should do the same.

Overall design (all in core):

• Metadata support for defining vector columns, similarity functions and vector search indexes. Support the standard similarity functions and index types out of the box, but allow extensibility too (see the MEVD example).
• Unified querying capability over vectors in core (so for all providers)
• Make sure we allow projecting out the similarity scores in some way.

Also keep full text search in mind, and hybrid.

Note: this would ideally also include EF taking care of embedding generation (#34387), so that users can have simple e.g. properties that EF automatically generates as embeddings.

[frankbuckley]

Note: this would ideally also include EF taking care of embedding generation (#34387), so that users can have simple e.g. properties that EF automatically generates as embeddings.

FWIW, having prototyped a vector search implementation with the new SQL Server features and EF Core, I think the recommendation to keep vectors in separate tables from the entities to which they relate is good advice:

• https://devblogs.microsoft.com/azure-sql/efficiently-and-elegantly-modeling-embeddings-in-azure-sql-and-sql-server/#the-right-way:-one-embedding-table-per-text-field

Most of the time you do not want vectors returned as part of the entities.

An 'automated' integration with IEmbeddingGenerator might want to support that scenario.

That said, embedding generation slows updates. We've ended up keeping vectors 'behind the scenes' in terms of the domain model and made index generation a separate process - periodically scanning for where documents have been updated then chunking and indexing accordingly.

Great to see support coming to EF though!

[roji]

@frankbuckley thanks for the feedback/thoughts.

FWIW, having prototyped a vector search implementation with the new SQL Server features and EF Core, I think the recommendation to keep vectors in separate tables from the entities to which they relate is good advice

Sure, but I don't think EF should be opinionated about something like this (or provide a higher-level feature); in other words, users should be able to model separate tables for embeddings as usual, just like EF already allows modeling different tables regardless of embeddings. While the advice in the post above absolutely makes sense in many/most cases, it won't make sense for all of them, and users should be able to have their embedding columns wherever fits their application.

[roji]

Notes from design discussion:

• We'll design a concept of a vector property in EF's model/metadata; this will be done in core, applying to all providers. This metadata will include vector-specific facets such as the dimensions; it may make sense to have common metadata for similarity function, index type, or it may be better to leave these ones provider-specific - TBD.
  • We most likely don't want to impose any specific CLR types here. The baseline support should be for arrays (float[], Half[]), ReadOnlyMemory<T> and Embedding<T>, but providers that want to should be to also accept their own custom types (e.g. SqlVector<T>). See #37025 for a bit of context.
• We'll design a unified LINQ search API for vector search, as described above. Cross-database research will be needed to make sure we design something that will work across databases (SQL Server, PostgreSQL, Cosmos...).
• On SQL Server, the unified search API should map to VECTOR_DISTANCE(), but if a vector index (#37281) is defined, VECTOR_SEARCH() should be used instead.
  • Note that VECTOR_SEARCH() is a TVF, whereas our unified API will mostly likely not be. So a possible non-trivial query transformation will be necessary there.
• Consider whether we want to obsolete the existing, provider-specific APIs, or keep them. Consider also the API naming to minimize ambiguity/confusion between the two.
• We should not allow vector properties inside relational JSON, as relational databases don't typically support that (document databases obviously do). However, table splitting/sharing would ideally support this (complex, possibly also owned), though this is likely something we can defer for later.
• When preparing the design, I'd recommend getting familiar with Microsoft.Extensions.VectorData, which already went through the exercise of designing a database-agnostic vector layer.