Introduce vector field, vector query and rescoring based on them

[mayya-sharipova]

Introduce a new field of type vector on which vector calculations can be done during rescoring phase

PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "my_feature": {
          "type": "vector"   
      }
    }
  }
}

Indexing

Allow only a single value per document
Allow to index both dense and sparse vectors?

Dense form:

PUT my_index/_doc/1
{
  "my_feature":   [11.5, 10.4, 23.0]
}

Sparse form (represented as list of dimension names and values for corresponding dimensions):

PUT my_index/_doc/1
{
  "my_feature": {"1": 11.5, "5": 10.5,  "101": 23.0}
}

Query and Rescoring

Introduce a special type of vector query:

"vector" : {
   "field" : "my_feature",
    "query_vector": {"1": 3, "5": 10.5,  "101": 12}
}

This query can only be used in the rescoring context.
This query produces a score for every document in the rescoring context in the following way:

1. If a document doesn't have a vector value for field, 0 value will be returned
2. If a document does have a vector value for field : doc_vector, the cosine similarity between doc_vector and query_vector is calculated:
   dotProduct(doc_vector, query_vector) / (sqrt(doc_vector) * sqrt(query_vector))

POST /_search
{
   "query" : {"<user-query>"},
   "rescore" : {
      "window_size" : 50,
      "query" : {
         "rescore_query" : {
            "vector" : {
               "field" : "my_feature",
               "query_vector": {"1": 3, "5": 10.5,  "101": 12}
            }
         }
      }
   }
}

Internal encoding

1. Encoding of vectors:
   Internally both dense and sparse vectors are encoded as sorted hash?
   Thus dense array is transformed:
   [4, 12] -> {0: 4, 1: 12}
   Keys are sorted, so we can iterate over them instead of calculating hash

2. What should be values in vectors?

   • floats?
   • smaller than floats? (lost some precision here, but less index size)

3. Vectors are encoded as binaries.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jpountz]

This query can only be used in the rescoring context.

If we want to enforce this, then it might be easier to have a rescorer rather than a query (today we only have one rescore implementation: QueryRescorer, but we can add more of them, see eg. https://github.com/elastic/elasticsearch/tree/master/plugins/examples/rescore). We might also want to give it a more explicit name like cosine_similarity?

[etienne1985]

Hi, commenting here on @mayya-sharipova 's invitation. Our use case is that we'd want to use ES to search for sentences that have similar meaning to the sentence in the query, based on each sentence having an embedding. Vectors would be dense. Dimensionality would be 100-300 most of the time presumably. Cosine similarity would be my starting point for computing the similarity of embeddings.

[james-daily]

Allow only a single value per document

Do you mean only one vector field per document or only one value for each field? It would be useful to allow more than one one vector field per document for testing different embeddings, dimensionalities, etc. Something like:

PUT my_index
{
  "mappings": {
    "_doc": {
      "properties": {
        "GloVe": {
          "type": "vector"   
      },
        "word2vec": {
          "type": "vector"   
      }
    }
  }
}

[mayya-sharipova]

@james-daily Thanks for your feedback, James. Sorry, for a single value per document, we meant a single value per field, so it would be possible to have several vector fields.

[djptek]

Have you considered Manhattan distance as a cheaper alternative in terms of processing? Though this will not deliver the same result it can be comparable in terms of ranking vectors while delivering higher throughput than euclidian/cosine

[jtibshirani]

In case it’s useful, here’s another datapoint from @gangeli, who also expressed interest in the feature:

• Their use case also involves retrieving sentences or short paragraphs. Both the query and documents would be modelled using a sentence embedding (based on an RNN).
• Vectors are dense and can have from 50 - 1000 dimensions, but are concentrated in the 200 - 300 range.
• Ideally, cosine similarity would be applied to all documents when scoring (as opposed to just during a rescoring phase). In their use case, sentence retrieval is a component of a fairly general NLP pipeline, and they rely strongly on these sentence embeddings to understand synonyms/ textual similarity.

[mayya-sharipova]

@djp-search thanks for a suggestion, we will study Manhattan distance

@jtibshirani thanks for another use-case

[softwaredoug]

Are there plans to use this to control matching as well? Such as filter in/out based on proximity (maybe some kind of distance) to a point being queried? Then it would be applicable outside a rescoring context

[mayya-sharipova]

@softwaredoug We are still debating if we should use this field for matching, as it may make queries slow. For now the plan is to introduce two functions cosineSimilarity and dotProduct as a part of script score query. The idea is that these functions will be used for scoring after the match is already done.