[FEATURE] Support hybrid KNN and search queries

[alokkhs]

Is your feature request related to a problem?

OpenSearch does not allow hybrid KNN and search queries (specifying both query and knn search fields together). The search results are based either on KNN or default search, but there is no way in-built way to combine the scores from the two approaches, and their scores are on different scales. This requires custom solutions/models on top to be able to leverage both KNN and search together.

What solution would you like?

Elasticsearch 8.4 supports hybrid retrieval by providing both the knn option and a query: https://www.elastic.co/guide/en/elasticsearch/reference/8.4/knn-search.html#_combine_approximate_knn_with_other_features
We would like to use this feature on OpenSearch for search improvements.

What alternatives have you considered?

Using an additional ranking model on top of OpenSearch can help to some extent, but it would limit the search space, and increases complexity.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[dblock]

Moving this to k-nn.

[jmazanec15]

Hi @alokkhs

Scores on different scales

Yes this is a difficult problem. I think for this results' scores would need to be normalized and then the results could be combined. However, this is not possible from the context of the query dsl.

no in-built way to combine the scores from the two approaches

One approach for this in current query dsl is to use a boolean query with 2 should clauses, where one is a k-NN query. The effect will be to add the scores of the 2 queries together.

Overall, a more general solution may be needed for combining results from different queries. However, this would likely exist outside of the k-NN query/plugin.

[jmazanec15]

No activity closing