[META] Score Combination and Normalization for Semantics Search. Score Normalization for k-NN and BM25

[vamshin]

Is your feature request related to a problem?

BM25 works well in exact match use cases and k-NN score works well in understanding context and getting relevant documents. It is important to get benefits from both of these relevancy mechanisms and one could achieve by combining these scores. One caveat is scores are on different scales and hence some kind of normalization is required.

Older Issues and Discussions:

1. [FEATURE] Hybrid search using keyword matching and kNN k-NN#717
2. Discussion on combination of result sets from different query types OpenSearch#4557
3. Science Benchmarks: https://opensearch.org/blog/semantic-science-benchmarks/

Tasks

High Level Tasks:

• Create RFC for the Feature Request. [RFC] High Level Approach and Design For Normalization and Score Combination #126
• Build POC for the Feature.
• Low Level Design of various components.
  • API Design [RFC] Improved Hybrid Search relevancy by Normalization and Score Combination Feature API Design LLD #244
  • Search Phase Results Processor: [RFC]: Search Phase Results Processor  #152
  • Different Normalization and Score combination Methods: [FEATURE] Provide way of defining methods for score normalization and combination in scope of Hybrid search #228
  • New Doc Collector For Query Clause: [FEATURE] Provide way of defining methods for score normalization and combination in scope of Hybrid search #228
  • LLD for New Query Phase Searcher: [FEATURE] Provide way of defining methods for score normalization and combination in scope of Hybrid search #228
• Validate Accuracy of the solution.
• Perf Testing and release tasks.
• Documentation issue for Normalization and Score combination feature: [DOC] Score Normalization and Combination for Semantic Search documentation-website#4351

Community Requests

1. https://forum.opensearch.org/t/normalisation-in-hybrid-search/12996

[navneet1v]

RFC created with HLD: #126

[martin-gaievski]

RFC with design for Query: #174
Created new Feature issue for Query: #175

[DarshitChanpura]

@vamshin Should this issue be moved to 2.11?

[vamshin]

@DarshitChanpura we are releasing this feature for 2.10

[navneet1v]

@martin-gaievski can we add the perf results on this issue and then resolve this github issue.

[martin-gaievski]

Summary for performance benchmark

dataset	p50 bool (baseline)	p50 hybrid	p50 difference, ms	p90 bool (baseline)	p90 hybrid	p90 difference, ms	p99 bool (baseline)	p99 hybrid	p99 difference, ms
nfcorpus	35.1	37	1.9	53	54	1	60.8	62.3	1.5
trec-covid	58.1	61.6	3.5	66.4	70	3.6	70.5	74.6	4.1
scidocs	54.9	57	2.1	66.4	68.6	2.2	81.3	83.2	1.9
quora	61	69	8	69	78	9	73.4	84	10.6
amazon esci	49	50	1	58	59.4	1.4	67	70	3
dbpedia	100.8	107.7	6.9	117	130.9	13.9	129.8	150.2	20.4
fiqa	53.9	56.9	3	61.9	65	3.1	64	67.7	3.7
% change vs. Boolean			6.40%			6.96%			8.27%

We took a bool query with neural and match sub-queries as a baseline.

For the cluster configuration, we used 3 data nodes of type “r5.8xlarge” and 1 leader node of type “c4.2xlarge”. All scripts that we use for benchmarks can be found in this repository.