Skip to content

[Approximation Framework] Extending approximation to top-level term queries on numeric fields #18679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: main
Choose a base branch
from
+387 −73
Open

[Approximation Framework] Extending approximation to top-level term queries on numeric fields #18679

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
d1ce965
Added approximation for term queries on numeric fields
sawansri Jun 30, 2025
dc161bc
Added basic ApproximateTermQuery implementation and tests
sawansri Jul 1, 2025
18d5180
Refactored approximated term queries to directly use ApproximatePoint…
sawansri Jul 2, 2025
74ff757
Added different numeric types to ApproximateTermQueryTests
sawansri Jul 2, 2025
83f47e3
update termQuery to properly wrap approximation
sawansri Jul 3, 2025
0fca1da
Updated tests in ScaledFloatFieldTypeTests to reflect new approximati…
sawansri Jul 3, 2025
1c21092
added ApproximateScoreQuery to valid term query types in TermQueryBui…
sawansri Jul 4, 2025
ef56a28
Refactored NumberFieldTypeTests to properly compare term queries
sawansri Jul 4, 2025
03936ee
Refactored approximation tests for term queries to be parameterized
sawansri Jul 6, 2025
5fce0a8
fix tests to be deterministic and term query sort edge case
sawansri Aug 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Prevent shard initialization failure due to streaming consumer errors ([#18877](https://github.com/opensearch-project/OpenSearch/pull/18877))
- APIs for stream transport and new stream-based search api action ([#18722](https://github.com/opensearch-project/OpenSearch/pull/18722))
- Added the core process for warming merged segments in remote-store enabled domains ([#18683](https://github.com/opensearch-project/OpenSearch/pull/18683))
- [Approximation Framework] Extended approximation to top-level term queries on numeric fields ([#18679](https://github.com/opensearch-project/OpenSearch/pull/18679))

### Changed
- Update Subject interface to use CheckedRunnable ([#18570](https://github.com/opensearch-project/OpenSearch/issues/18570))
Expand Down
16 changes: 15 additions & 1 deletion ...es/mapper-extras/src/test/java/org/opensearch/index/mapper/ScaledFloatFieldTypeTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
import org.opensearch.index.fielddata.IndexNumericFieldData;
import org.opensearch.index.fielddata.LeafNumericFieldData;
import org.opensearch.index.fielddata.SortedNumericDoubleValues;
import org.opensearch.search.approximate.ApproximatePointRangeQuery;
import org.opensearch.search.approximate.ApproximateScoreQuery;

import java.io.IOException;
Expand All @@ -68,7 +69,17 @@ public void testTermQuery() {
long scaledValue = Math.round(value * ft.getScalingFactor());
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery("scaled_float", scaledValue);
Query query = new IndexOrDocValuesQuery(LongPoint.newExactQuery("scaled_float", scaledValue), dvQuery);
assertEquals(query, ft.termQuery(value, null));
Query expectedQuery = new ApproximateScoreQuery(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we dont change anything in ScaledFloatFieldMapper because the termQuery uses the NumberFieldMapper.NumberType.LONG.termQuery

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, ScaledFloatFieldMapper implements a termQuery that uses NumberFieldMapper.NumberType.LONG.termQuery() which would fall under approximation.

query,
new ApproximatePointRangeQuery(
"scaled_float",
LongPoint.pack(new long[] { scaledValue }).bytes,
LongPoint.pack(new long[] { scaledValue }).bytes,
1,
ApproximatePointRangeQuery.LONG_FORMAT
)
);
assertEquals(expectedQuery, ft.termQuery(value, null));
}

public void testTermsQuery() {
Expand Down Expand Up @@ -128,6 +139,9 @@ public void testRangeQuery() throws IOException {
MOCK_QSC
);
Query scaledFloatQ = ft.rangeQuery(l, u, includeLower, includeUpper, MOCK_QSC);
assertTrue(scaledFloatQ instanceof ApproximateScoreQuery);
Query approximationQuery = ((ApproximateScoreQuery) scaledFloatQ).getApproximationQuery();
assertTrue(approximationQuery instanceof ApproximatePointRangeQuery);
assertEquals(searcher.count(doubleQ), searcher.count(scaledFloatQ));
}
IOUtils.close(reader, dir);
Expand Down
178 changes: 134 additions & 44 deletions server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,15 +298,34 @@ public Float parse(XContentParser parser, boolean coerce) throws IOException {
@Override
public Query termQuery(String field, Object value, boolean hasDocValues, boolean isSearchable) {
float v = parse(value, false);
if (isSearchable && hasDocValues) {
Query query = HalfFloatPoint.newExactQuery(field, v);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, HalfFloatPoint.halfFloatToSortableShort(v));
return new IndexOrDocValuesQuery(query, dvQuery);
}
Query dvQuery = null;
if (hasDocValues) {
return SortedNumericDocValuesField.newSlowExactQuery(field, HalfFloatPoint.halfFloatToSortableShort(v));
dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, HalfFloatPoint.halfFloatToSortableShort(v));
}
if (isSearchable) {
Query pointQuery = HalfFloatPoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}

// Use this instead of HalfFloatPoint's pack until Lucene 10.3.0
byte[] exactPoint = NumberType.HALF_FLOAT.encodePoint(v);

return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(
field,
exactPoint,
exactPoint,
new float[] { v }.length,
ApproximatePointRangeQuery.HALF_FLOAT_FORMAT
)
);
}
return HalfFloatPoint.newExactQuery(field, v);
return dvQuery;
}

@Override
Expand Down Expand Up @@ -474,15 +493,30 @@ public Float parse(XContentParser parser, boolean coerce) throws IOException {
@Override
public Query termQuery(String field, Object value, boolean hasDocValues, boolean isSearchable) {
float v = parse(value, false);
if (isSearchable && hasDocValues) {
Query query = FloatPoint.newExactQuery(field, v);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.floatToSortableInt(v));
return new IndexOrDocValuesQuery(query, dvQuery);
}
Query dvQuery = null;
if (hasDocValues) {
return SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.floatToSortableInt(v));
dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.floatToSortableInt(v));
}
return FloatPoint.newExactQuery(field, v);
if (isSearchable) {
Query pointQuery = FloatPoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}
return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(
field,
FloatPoint.pack(new float[] { v }).bytes,
FloatPoint.pack(new float[] { v }).bytes,
new float[] { v }.length,
ApproximatePointRangeQuery.FLOAT_FORMAT
)
);
}
return dvQuery;
}

@Override
Expand Down Expand Up @@ -641,15 +675,30 @@ public Double parse(XContentParser parser, boolean coerce) throws IOException {
@Override
public Query termQuery(String field, Object value, boolean hasDocValues, boolean isSearchable) {
double v = parse(value, false);
if (isSearchable && hasDocValues) {
Query query = DoublePoint.newExactQuery(field, v);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.doubleToSortableLong(v));
return new IndexOrDocValuesQuery(query, dvQuery);
}
Query dvQuery = null;
if (hasDocValues) {
return SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.doubleToSortableLong(v));
dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, NumericUtils.doubleToSortableLong(v));
}
if (isSearchable) {
Query pointQuery = DoublePoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}
return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(
field,
DoublePoint.pack(new double[] { v }).bytes,
DoublePoint.pack(new double[] { v }).bytes,
new double[] { v }.length,
ApproximatePointRangeQuery.DOUBLE_FORMAT
)
);
}
return DoublePoint.newExactQuery(field, v);
return dvQuery;
}

@Override
Expand Down Expand Up @@ -969,15 +1018,30 @@ public Query termQuery(String field, Object value, boolean hasDocValues, boolean
return Queries.newMatchNoDocsQuery("Value [" + value + "] has a decimal part");
}
int v = parse(value, true);
if (isSearchable && hasDocValues) {
Query query = IntPoint.newExactQuery(field, v);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, v);
return new IndexOrDocValuesQuery(query, dvQuery);
}
Query dvQuery = null;
if (hasDocValues) {
return SortedNumericDocValuesField.newSlowExactQuery(field, v);
dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, v);
}
if (isSearchable) {
Query pointQuery = IntPoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}
return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(
field,
IntPoint.pack(new int[] { v }).bytes,
IntPoint.pack(new int[] { v }).bytes,
new int[] { v }.length,
ApproximatePointRangeQuery.INT_FORMAT
)
);
}
return IntPoint.newExactQuery(field, v);
return dvQuery;
}

@Override
Expand Down Expand Up @@ -1155,17 +1219,30 @@ public Query termQuery(String field, Object value, boolean hasDocValues, boolean
return Queries.newMatchNoDocsQuery("Value [" + value + "] has a decimal part");
}
long v = parse(value, true);
if (isSearchable && hasDocValues) {
Query query = LongPoint.newExactQuery(field, v);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, v);
return new IndexOrDocValuesQuery(query, dvQuery);

}
Query dvQuery = null;
if (hasDocValues) {
return SortedNumericDocValuesField.newSlowExactQuery(field, v);

dvQuery = SortedNumericDocValuesField.newSlowExactQuery(field, v);
}
return LongPoint.newExactQuery(field, v);
if (isSearchable) {
Query pointQuery = LongPoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}
return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(
field,
LongPoint.pack(new long[] { v }).bytes,
LongPoint.pack(new long[] { v }).bytes,
new long[] { v }.length,
ApproximatePointRangeQuery.LONG_FORMAT
)
);
}
return dvQuery;
}

@Override
Expand Down Expand Up @@ -1297,15 +1374,28 @@ public Query termQuery(String field, Object value, boolean hasDocValues, boolean
return Queries.newMatchNoDocsQuery("Value [" + value + "] has a decimal part");
}
BigInteger v = parse(value, true);
if (isSearchable && hasDocValues) {
Query query = BigIntegerPoint.newExactQuery(field, v);
Query dvQuery = SortedUnsignedLongDocValuesSetQuery.newSlowExactQuery(field, v);
return new IndexOrDocValuesQuery(query, dvQuery);
}
Query dvQuery = null;
if (hasDocValues) {
return SortedUnsignedLongDocValuesSetQuery.newSlowExactQuery(field, v);
dvQuery = SortedUnsignedLongDocValuesSetQuery.newSlowExactQuery(field, v);
}
return BigIntegerPoint.newExactQuery(field, v);
if (isSearchable) {
Query pointQuery = BigIntegerPoint.newExactQuery(field, v);
Query query;
if (dvQuery != null) {
query = new IndexOrDocValuesQuery(pointQuery, dvQuery);
} else {
query = pointQuery;
}

// Until Lucene 10.3.0
byte[] exactPoint = NumberType.UNSIGNED_LONG.encodePoint(v);

return new ApproximateScoreQuery(
query,
new ApproximatePointRangeQuery(field, exactPoint, exactPoint, 1, ApproximatePointRangeQuery.UNSIGNED_LONG_FORMAT)
);
}
return dvQuery;
}

@Override
Expand Down
8 changes: 7 additions & 1 deletion server/src/main/java/org/opensearch/search/approximate/ApproximatePointRangeQuery.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.opensearch.search.sort.SortOrder;

import java.io.IOException;
import java.util.Arrays;
import java.util.Objects;
import java.util.function.Function;

Expand Down Expand Up @@ -392,7 +393,12 @@ public long cost() {

@Override
public Scorer get(long leadCost) throws IOException {
intersectRight(values.getPointTree(), visitor, docCount);
// Force term queries with desc sort through intersectLeft
if (Arrays.equals(pointRangeQuery.getLowerPoint(), pointRangeQuery.getUpperPoint())) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it work with ?

curl -X POST "http://localhost:9200/logs-*/_search" -H "Content-Type: application/json" -d '
{
  "query": {
    "term": {
      "status": {
        "value": 200
      }
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
' | jq '.'

Copy link
Member

@prudhvigodithi prudhvigodithi Aug 5, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Today we can run term with sort, I want to make sure the results does not break,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a good change

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it work with ?

curl -X POST "http://localhost:9200/logs-*/_search" -H "Content-Type: application/json" -d '
{
  "query": {
    "term": {
      "status": {
        "value": 200
      }
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
' | jq '.'

This change doesn't affect term queries where the primary sort field is different from the term query's field. In ApproximatePointRangeQuery's canApproximate, this statement gets executed.

if (!primarySortField.fieldName().equals(pointRangeQuery.getField())) {
                    return false;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there is a singular unique point value in a numeric term query and the tiebreaker ends up being the docID, using intersectLeft results in the same top-k results as intersectRight.

I'm also fine in disabling approximation for this specific case (term query with descending sort) due to performance concerns (queue has more comparisons to make) until intersectRight logic can be modified to return accurate hits for these kinds of queries.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is risky, can leave out this change and create an issue for it? I'm to merge in incremental changes, given there's enough test coverage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that's what I was thinking as well. For the time being, we could disable approximation and send desc sorts on top level term queries through Lucene while this edge case is being sorted out.

intersectLeft(values.getPointTree(), visitor, docCount);
} else {
intersectRight(values.getPointTree(), visitor, docCount);
}
DocIdSetIterator iterator = result.build().iterator();
return new ConstantScoreScorer(score(), scoreMode, iterator);
}
Expand Down
37 changes: 32 additions & 5 deletions server/src/test/java/org/opensearch/index/mapper/NumberFieldTypeTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,8 +181,19 @@ private static MappedFieldType unsearchable() {
public void testTermQuery() {
MappedFieldType ft = new NumberFieldMapper.NumberFieldType("field", NumberFieldMapper.NumberType.LONG);
Query dvQuery = SortedNumericDocValuesField.newSlowExactQuery("field", 42);
Query query = new IndexOrDocValuesQuery(LongPoint.newExactQuery("field", 42), dvQuery);
assertEquals(query, ft.termQuery("42", null));
Query pointQuery = LongPoint.newExactQuery("field", 42);
Query indexOrDocValuesQuery = new IndexOrDocValuesQuery(pointQuery, dvQuery);
Query approximateQuery = new ApproximateScoreQuery(
indexOrDocValuesQuery,
new ApproximatePointRangeQuery(
"field",
LongPoint.pack(new long[] { 42 }).bytes,
LongPoint.pack(new long[] { 42 }).bytes,
1,
ApproximatePointRangeQuery.LONG_FORMAT
)
);
assertEquals(approximateQuery, ft.termQuery("42", null));

MappedFieldType unsearchable = unsearchable();
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> unsearchable.termQuery("42", null));
Expand Down Expand Up @@ -641,10 +652,26 @@ public void testNegativeZero() {
NumberType.HALF_FLOAT.rangeQuery("field", null, +0f, true, false, false, true, MOCK_QSC)
);

assertFalse(NumberType.DOUBLE.termQuery("field", -0d, true, true).equals(NumberType.DOUBLE.termQuery("field", +0d, true, true)));
assertFalse(NumberType.FLOAT.termQuery("field", -0f, true, true).equals(NumberType.FLOAT.termQuery("field", +0f, true, true)));
// For term queries, we need to extract the original query from the ApproximateScoreQuery
Query negativeZeroDouble = NumberType.DOUBLE.termQuery("field", -0d, true, true);
Query positiveZeroDouble = NumberType.DOUBLE.termQuery("field", +0d, true, true);
assertFalse(
((ApproximateScoreQuery) negativeZeroDouble).getOriginalQuery()
.equals(((ApproximateScoreQuery) positiveZeroDouble).getOriginalQuery())
);

Query negativeZeroFloat = NumberType.FLOAT.termQuery("field", -0f, true, true);
Query positiveZeroFloat = NumberType.FLOAT.termQuery("field", +0f, true, true);
assertFalse(
((ApproximateScoreQuery) negativeZeroFloat).getOriginalQuery()
.equals(((ApproximateScoreQuery) positiveZeroFloat).getOriginalQuery())
);

Query negativeZeroHalfFloat = NumberType.HALF_FLOAT.termQuery("field", -0f, true, true);
Query positiveZeroHalfFloat = NumberType.HALF_FLOAT.termQuery("field", +0f, true, true);
assertFalse(
NumberType.HALF_FLOAT.termQuery("field", -0f, true, true).equals(NumberType.HALF_FLOAT.termQuery("field", +0f, true, true))
((ApproximateScoreQuery) negativeZeroHalfFloat).getOriginalQuery()
.equals(((ApproximateScoreQuery) positiveZeroHalfFloat).getOriginalQuery())
);
}

Expand Down
2 changes: 2 additions & 0 deletions server/src/test/java/org/opensearch/index/query/TermQueryBuilderTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
import org.opensearch.common.util.io.IOUtils;
import org.opensearch.core.common.ParsingException;
import org.opensearch.index.mapper.MappedFieldType;
import org.opensearch.search.approximate.ApproximateScoreQuery;

import java.io.IOException;

Expand Down Expand Up @@ -119,6 +120,7 @@ protected void doAssertLuceneQuery(TermQueryBuilder queryBuilder, Query query, Q
assertThat(
query,
either(instanceOf(TermQuery.class)).or(instanceOf(PointRangeQuery.class))
.or(instanceOf(ApproximateScoreQuery.class))
.or(instanceOf(MatchNoDocsQuery.class))
.or(instanceOf(AutomatonQuery.class))
.or(instanceOf(IndexOrDocValuesQuery.class))
Expand Down
Loading
Loading