Add option to normalize vector distances on query

[rbs333]

This pr accomplishes 2 goals:

1. Add an option for users to easily get back a similarity value between 0 and 1 that they might expect to compare against other vector dbs.
2. Fix the current bug that distance_threshold is validated to be between 0 and 1 when in reality it can take values between 0 and 2.

Note: after much careful thought I believe it is best that for 0.5.0 we do not start enforcing all distance_thresholds between 0 and 1 and move to this option as default behavior. Ideally this metric would be consistent throughout our code and I don't love supporting this flag but I think it provides the value that is scoped for this ticket while inflicting the least amount of pain and confusion.

Changes:

1. Adds the normalize_vector_distance flag to VectorQuery and VectorRangeQuery.

Behavior:

• If set to True it normalizes values returned from redis to a value between 0 and 1.
• For cosine similarity, it applies (2 - value)/2.
• For L2 distance, it applies normalization (1/(1+value)).
• For IP, it does nothing and throws a warning since normalized IP is cosine by definition.
• For VectorRangeQuery, if normalize_vector_distance=True the distance threshold is now validated to be between 0 and 1 and denormalized for execution against the database to make consistent.

2. Relaxes validation for semantic caching and routing to be between 0 and 2 fixing the current bug and aligning with how the database actually functions.

[rbs333]

Could add additional check logic to this function kept it simple stupid to start

[tylerhutcherson]

Nice @rbs333 good start. We do want to enable normalization regardless of distance metric.

1. So imagine a map between chosen/indexed distance metric and the normalization function by distance metric. So in that vein, we can rename the param to a generic normalize naming!
2. Redis maps cosine distance to 0-2 both in input and output. and I think our goal is to normalize it to 0-1 instead (which is how most users think of and use our implementation anyways because 0-2 is not well documented on redis docs.
3. there are some places where we currently (wrongly) enforce the distance threshold between 0-1, even though it's technically 0-2. (see semantic cache base class). this will be "correct" once you make this change as we can internally control that normalization param on the semantic cache query definitions (internals)

[rbs333]

Nice @rbs333 good start. We do want to enable normalization regardless of distance metric.

1. So imagine a map between chosen/indexed distance metric and the normalization function by distance metric. So in that vein, we can rename the param to a generic normalize naming!
2. Redis maps cosine distance to 0-2 both in input and output. and I think our goal is to normalize it to 0-1 instead (which is how most users think of and use our implementation anyways because 0-2 is not well documented on redis docs.
3. there are some places where we currently (wrongly) enforce the distance threshold between 0-1, even though it's technically 0-2. (see semantic cache base class). this will be "correct" once you make this change as we can internally control that normalization param on the semantic cache query definitions (internals)

1. Heard and can do.

Is inner product also bounded between 0-2 returned from redis? It is not documented well and if that is the case this formula is expressed wrong in the docs.

For euclidean, what normalization were you thinking? It's not obvious to me.

2. As implemented currently the output of the normalization is between 0 and 1.
3. Heard and will update.

[tylerhutcherson]

Nice @rbs333 good start. We do want to enable normalization regardless of distance metric.

1. So imagine a map between chosen/indexed distance metric and the normalization function by distance metric. So in that vein, we can rename the param to a generic normalize naming!
2. Redis maps cosine distance to 0-2 both in input and output. and I think our goal is to normalize it to 0-1 instead (which is how most users think of and use our implementation anyways because 0-2 is not well documented on redis docs.
3. there are some places where we currently (wrongly) enforce the distance threshold between 0-1, even though it's technically 0-2. (see semantic cache base class). this will be "correct" once you make this change as we can internally control that normalization param on the semantic cache query definitions (internals)

1. Heard and can do.

Is inner product also bounded between 0-2 returned from redis? It is not documented well and if that is the case this formula is expressed wrong in the docs.

For euclidean, what normalization were you thinking? It's not obvious to me.

2. As implemented currently the output of the normalization is between 0 and 1. 3. Heard and will update.

For euclidean we should be able to do 1/(1-raw_dist) since the raw distance value is unbounded on the upper limit.

[tylerhutcherson]

Ok looking solid, left a few comments. But unfortunately, we might need one more discussion on this as I am reading this code and thinking about 2 things:

• Given L2 is unbounded by definition, is there a scenario here where having this value normalized makes any sense in a use case? We might want to actually take a peek at 2-3 other vector db solutions to see if they document how they normalize these kinds of metrics.
• Secondly, and more importantly, if a user sets distance_threshold as a value between [x, y], wouldn't they expect the vector distance they get on the output to match this? Tricky

[rbs333]

Ok looking solid, left a few comments. But unfortunately, we might need one more discussion on this as I am reading this code and thinking about 2 things:

• Given L2 is unbounded by definition, is there a scenario here where having this value normalized makes any sense in a use case? We might want to actually take a peek at 2-3 other vector db solutions to see if they document how they normalize these kinds of metrics.
• Secondly, and more importantly, if a user sets distance_threshold as a value between [x, y], wouldn't they expect the vector distance they get on the output to match this? Tricky

@tylerhutcherson personally I think we should limit scope of this ticket to just normalize for cosine distance. The customer needs we are addressing, that I have actually witnessed people get confused by, is having an option for a vector similarity type output for cosine. I have not seen a single demo from a competitor or use case from a customer where they used euclidean distance and especially not normalized euclidean distance. I think we risk spinning our wheels on something nobody really cares about.

On the second point, yeah this was something I debated as well. Ideally I think we get to a point that for RangeQueries, at very least, normalization is the default option such that a distance_threshold between 0 and 1 always makes sense. The extra troubling thing is that this would work except for the cases involving the semantic_router right now we use aggregations to handle that with distance_threshold where we can't normalize ahead of time which makes things complicated.

[tylerhutcherson]

NICE

[tylerhutcherson]

nice!