Can we use top hits optimizations when sorting by a field?

[jpountz]

Currently optimizations for collection of top documents are only applicable when sorting by score or by a sort order that is congruent with the index-time sort. Given that fields are most of time both indexed and doc-valued, maybe we could optimize sorting in an even greater number of cases.

For instance imagine that a user searches for top documents matching query Q sorted by field @timestamp of type date in descending order. Maybe we could internally rewrite the query under the hood to SHOULD(LongDistanceFeatureQuery(field=@timestamp, origin=Long.MAX_VALUE)) FILTER(Q) and prepend _score to the sort: [ "_score", { "@timestamp": { "order": "desc" } } ].

There are a number of details to sort out, like we could only do that it the field is indexed and we would need to rewrite the produced TopFieldDocs so that they don't include a sort value for the score and maybe take the maximum value of the field rather than Long.MAX_VALUE to avoid defeating the optimization because most scores would be equal after casting to a float, but overall it looks to me like it could help optimize almost all sorted queries without requiring index-time decisions?

[elasticmachine]

Pinging @elastic/es-search

[mayya-sharipova]

@jpountz Do we want to expose LongDistanceFeatureQuery in elasticsearch and create a corresponding QueryBuilder for it? For example :

{
  "query": {
    "bool": {
      "filter": [
        {
          "match": {
            "content": "2016"
          }
        }
      ],
      "should": [
        {
          "long_distance_rank_feature": {
            "field": "timestamp",
            "boost": 2,
            "pivot" : 10,
            "origin" : 1550175141321
          }
        }
      ]
    }
  }
}

[jpountz]

I think we should (#33382). But for this particular issue the idea is to rewrite the search request under the hood to something that returns the same hits but runs faster.

[mayya-sharipova]

@jpountz thanks for reminding about #33382.

But for this particular issue the idea is to rewrite the search request under the hood to something that returns the same hits but runs faster.

Right, I was trying to do rewrite in SearchSourceBuilder::rewrite method, and noticed first we will need a QueryBuilder for LongDistanceFeatureQuery.

final long origin = (sort.order() == SortOrder.DESC) ? Long.MAX_VALUE : Long.MIN_VALUE;
final long pivotDistance = 5;  // ? not sure what to choose for pivot distance
Query query = LongPoint.newDistanceFeatureQuery(fieldName, 1, origin, pivotDistance);

BoolQueryBuilder rewritten = new BoolQueryBuilder();
rewritten.should();  // need a QueryBuilder for DistanceFeatureQuery to insert it here
rewritten.filter(queryBuilder); // filter for original query

I guess the plan is first to implement #33382

[jpountz]

do rewrite in SearchSourceBuilder::rewrite

In my opinion one challenge with this optimization is that we also need to modify the produced FieldDoc.fields to remove the first element. I suspect implementing this optimization in SearchSourceBuilder#rewrite is going to make it challenging to also modify the produced TopFieldDocs when appropriate. It might have downsides that I haven't though about, but my initial idea was to do it in QueryPhase#execute by modifying the query before execute runs, and modifying the produced top docs before they are set on the search context.

not sure what to choose for pivot distance

The tricky bit with origin and pivotDistance is that we should pick values that make it unlikely that different long values map to the same float score. We'd probably need to use either the min or max value on the shard as the origin, and something in the order of (max-min)/2 for the pivot value.

I guess the plan is first to implement #33382

That works for me if that makes things easier.

[dvisztempacct]

I was linked to this issue by very helpful people at the elasticon ama booth.

Do I understand correctly that this enhancement would mean I could avoid the big sort-all-matching-documents operation when I want to sort by the value from a single field?

That would be a massive performance improvement for me!

[jpountz]

@dvisztempacct You got it right.

[dvisztempacct]

@jpountz thank you! I'm very much looking forward to this!