Preview Vector Set API support

[kevin-montrose]

Implement a small subset of Vector Set functionality, as a base for future full support.

This PR:

• Uses DiskANN to do the actual vector operations
  • While not available yet, latest DiskANN (and the diskann-garnet integration package) should be OSS'd soon
  • The expectation is, even if reviewed and approved, this PR will not be merged until diskann-garnet is available in nuget.org (and ideally the source is also available on GitHub)
• Introduces a notion of namespaces to Tsavorite, which is used to "hide" data from other commands
• Adds a VectorSet bit to RecordInfo to allow distinguishing Vector Sets from other keys
• Implements a subset of VADD, VREM, VEMB, and VSIM
• Adds two extensions to these Redis commands:
  • XB8 - which allows passing byte-values without conversion, joining FP32 (which is used for little-endian floats)
  • XPREQ8 - a quantization option which takes in pre-quantized vectors, meant for use with XB8
• Recovery, replication, and migration for Vector Sets
• Hides all this functionality behind EnableVectorSetPreview/--enable-vector-set-preview

A more complete write up of the design, justifications, and remaining work is in vector-set.md.

---

There is still a lot of work to be done on Vector Sets, but this PR is at a point where it's functional enough to be played with - and there's merit to merging it so other work (Store V2, multi-threaded replication, etc.) isn't complicated.

The "big" things (besides commands and options) that are missing are:

• Non-XPREQ8 quantizers - implementation here is on DiskANN
• Variable length vector ids - likewise, support is coming in DiskANN, though some Garnet work will also be needed

[badrishc]

This all makes sense. In storage-v2, we have the notion of a LogRecord record format, that already includes optional fields such as ETag and Expiration. Adding a namespace would be analogous to this, with a couple of differences:

• We use one RecordInfo bit to indicate whether the record has a namespace field
• If true, then the NameSpace byte (or larger if needed) exists as part of the record, roughly: <RecordInfo, key, value, etag?, expiration?, namespace?, ...>
• Tsavorite APIs are adjusted to accept the optional namespace (in addition to key/expiration/etag).
• All hash codes and key equalities in Tsavorite would need to incorporate the namespace in their computations.
• Certain namespaces (e.g., namespace numbers starting with "11") would be reserved for vector-set usage. Within that sub-namespace, the vector-sets can partition bits as they want.

TBD: how to handle larger namespace names in the same framework (e.g., "/user/foo/"), and is that useful/necessary to make as a first-class citizen versus users directly incorporating in the actual key.

[prvyk]

Adding a namespace would be analogous to this, with a couple of differences:

...

TBD: how to handle larger namespace names in the same framework (e.g., "/user/foo/"), and is that useful/necessary to make as a first-class citizen versus users directly incorporating in the actual key.

Namespaces can be used for a lot of things, like replacing the current numbered database implementation with something that is supported cluster-wide as in valkey (In that case they end up in the same AOF file I guess?). Also, quite a few RESP-accepting DBs have a namespace implementation of sorts, e.g. kvrocks. I had an idea of combining ACLs and database numbers (IMHO, this would be usually better than redis's prefix ACLs, because it doesn't require client to cooperate), and namespaces would be just as natural here.

[vazois]

I might have missed it, but we need a way to cancel these tasks in the event the replication stream breaks (i.e. failover).

[kevin-montrose]

What's the signal for that? Looking at TryFailover and nothing is jumping out - seems like AofProcessor (and its associated RespServerSession) is kept live as long as the ClusterProvider is.

Since the AOF stream would be interrupted during failover, I'm not sure aborting the task is actually necessary - they'll just stay pending until/unless a new AOF stream is received.

[vazois]

There is a cancellationToken in ReplicaReplayTask.cs. This was not passed down to ProcessAofRecordInternal because we didn't have any background tasks in the AofProcessor.cs (as far as I know).
So, you may use that.

[tiagonapoli]

Just for my understanding - I don't fully understand the Tsavorite value sizing yet

Could we possibly have an unbounded size for the value stored on this case? For each time we add or delete to the list of deleting vector sets, GetRMWModifiedValueLength is saying that the new list size will be value.Length + abs(input.WriteDesiredSize) - strictly increasing - so if Tsavorite actually allocates the new size and that's unused, but Tsavorite doesn't know it, the next time we call it, we will increase again and again?