Add bitmap64 query support

[divyaruhil]

Description

Feature Request :- #20597

Related Issues

Resolves #20597

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This pull request introduces 64-bit bitmap query support to OpenSearch by adding two new Lucene Query implementations (for index and doc values), integrating them into NumberFieldMapper for LONG numeric fields, and providing comprehensive test coverage for both query types.

Changes

Cohort / File(s)	Summary
Mapper Integration server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java	Added bitmapQuery() override for LONG type that deserializes 64-bit Roaring bitmap from BytesArray and returns appropriate query type (IndexQuery, DocValuesQuery, or IndexOrDocValuesQuery) based on searchability and doc values availability.
64-bit Bitmap Index Query server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java	New Query implementation for matching documents using 64-bit Roaring bitmap via point value intersection; includes weight creation with MergePointVisitor for efficient intersection, caching, equality, and memory accounting.
64-bit Bitmap DocValues Query server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java	New Query implementation for matching documents using 64-bit Roaring bitmap via sorted numeric doc values; includes TwoPhaseIterator with optimized singleton path, weight creation, caching, equality, and memory accounting.
Query Tests server/src/test/java/org/opensearch/search/query/Bitmap64DocValuesQueryTests.java, server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java	Test suites for both query types covering single/multi-value documents, empty bitmaps, range optimization, large values, null validation, and random document/query scenarios.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Mapper as NumberFieldMapper
    participant Deserializer as Bitmap Deserializer
    participant QueryFactory as Query Type Selector
    participant IndexQuery as Bitmap64IndexQuery
    participant DocValuesQuery as Bitmap64DocValuesQuery
    participant Searcher as IndexSearcher

    Client->>Mapper: bitmapQuery(field, BytesArray, isSearchable, hasDocValues)
    Mapper->>Deserializer: deserialize 64-bit Roaring bitmap
    Deserializer-->>Mapper: Roaring64NavigableMap
    
    alt isSearchable && hasDocValues
        Mapper->>QueryFactory: Create IndexOrDocValuesQuery
        QueryFactory->>IndexQuery: new Bitmap64IndexQuery(field, bitmap)
        QueryFactory->>DocValuesQuery: new Bitmap64DocValuesQuery(field, bitmap)
        QueryFactory-->>Mapper: IndexOrDocValuesQuery(indexQuery, docValuesQuery)
    else isSearchable
        Mapper->>IndexQuery: new Bitmap64IndexQuery(field, bitmap)
        IndexQuery-->>Mapper: IndexQuery
    else hasDocValues
        Mapper->>DocValuesQuery: new Bitmap64DocValuesQuery(field, bitmap)
        DocValuesQuery-->>Mapper: DocValuesQuery
    end
    
    Mapper-->>Client: Query
    Client->>Searcher: search(query)
    
    alt IndexQuery Path
        Searcher->>IndexQuery: createWeight()
        IndexQuery->>Searcher: PointValues ∩ Bitmap
        Searcher-->>Client: DocIdSet(matching docs)
    else DocValuesQuery Path
        Searcher->>DocValuesQuery: createWeight()
        DocValuesQuery->>Searcher: SortedNumericDocValues ∩ Bitmap
        Searcher-->>Client: DocIdSet(matching docs)
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Suggested labels

enhancement, Indexing & Search, lucene

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	❓ Inconclusive	The PR description references a feature request and resolves issue #20597, with tests included, but lacks detail on what the bitmap64 query feature actually does.	Add a detailed description of the bitmap64 query feature, its purpose, use cases, and how it works to improve clarity for reviewers.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'Add bitmap64 query support' directly and concisely describes the main change: implementing bitmap64 query functionality. It is clear, specific, and directly relevant to all the changes in the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

❌ Gradle check result for 0dcc129: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[coderabbitai]

Actionable comments posted: 4

🤖 Fix all issues with AI agents

In `@server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java`:
- Around line 1376-1399: The catch in NumberFieldMapper.bitmapQuery currently
catches Exception; change it to catch IOException (the declared exception from
deserializePortable) to avoid masking other errors—update the catch block in the
bitmapQuery method that deserializes into Roaring64NavigableMap to "catch
(IOException e)"; make the identical change in the INTEGER.bitmapQuery
deserialization catch as well; and add a short inline comment near both
bitmap64/bitmap32 usages (and the constructors for
Bitmap64IndexQuery/Bitmap64DocValuesQuery or where they are instantiated) noting
that the Roaring* bitmap is read-only after deserialization and must not be
modified by the query classes.

In
`@server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java`:
- Around line 1-3: Replace the short 3-line SPDX headers (or missing header) in
the new source and test files with the project's full 7-line header including
the OpenSearch contributor notice; specifically update the header comment at the
top of Bitmap64DocValuesQuery.java (class Bitmap64DocValuesQuery),
Bitmap64IndexQuery.java (class Bitmap64IndexQuery), and
Bitmap64IndexQueryTests.java (class Bitmap64IndexQueryTests) so they all start
with: "/*\n * SPDX-License-Identifier: Apache-2.0\n *\n * The OpenSearch
Contributors require contributions made to\n * this file be licensed under the
Apache-2.0 license or a\n * compatible open source license.\n */" ensuring exact
text and spacing to match existing project files.
- Around line 104-105: The code currently constructs a DefaultScorerSupplier
directly (e.g., in Bitmap64DocValuesQuery where you return new
DefaultScorerSupplier(scorer)); change these to use the fully-qualified nested
type Weight.DefaultScorerSupplier instead (return new
Weight.DefaultScorerSupplier(scorer)), and update all similar usages (e.g., in
BitmapDocValuesQuery, ShardSplittingQuery, etc.) so they reference
Weight.DefaultScorerSupplier rather than the unqualified DefaultScorerSupplier.

In
`@server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java`:
- Line 1: Add the missing SPDX license header to the top of the file for
Bitmap64IndexQueryTests: insert the same Apache-2.0 SPDX header block used in
other new files (for example Bitmap64DocValuesQueryTests) as the very first
lines of
server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java so
the file begins with the standard SPDX/Apache-2.0 header before the package
declaration and class definition.

🧹 Nitpick comments (7)

server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java (3)

29-29: Unused import: PeekableLongIterator is imported but never used.

Line 64 uses org.roaringbitmap.longlong.LongIterator directly. PeekableLongIterator (which extends LongIterator and adds advanceIfNeeded(long)) would actually be useful here — see next comment about advance() performance.

---

66-66: Inconsistent indentation: tab character mixed with spaces.

Line 66 uses a tab character for indentation while the rest of the file uses spaces.

---

141-208: MergePointVisitor is an inner class but could be static with the bitmap passed as a parameter.

MergePointVisitor is a non-static inner class that captures the enclosing Bitmap64IndexQuery instance solely to access bitmap. Making it static and passing bitmap to the constructor would make the dependency explicit and avoid retaining a reference to the entire query object.

That said, this mirrors the pattern used in the 32-bit BitmapIndexQuery, so it's consistent with the existing codebase.

server/src/test/java/org/opensearch/search/query/Bitmap64DocValuesQueryTests.java (2)

130-136: query.rewrite(null) relies on short-circuit before super.rewrite(null) is reached.

This works today because the empty-bitmap check returns early. If the implementation ever changes to invoke super before the check, this test will NPE. Consider creating a minimal IndexSearcher (e.g. from an empty directory) instead.

---

163-193: Consider adding a test with negative long values.

The current testLargeValues tests near Long.MAX_VALUE, which is great. However, there's no test for negative long values (e.g., Long.MIN_VALUE, negative ranges). Since LongPoint.encodeDimension uses an offset encoding that maps Long.MIN_VALUE → 0x0000... and Long.MAX_VALUE → 0xFFFF..., ensuring correct behavior across the signed boundary would strengthen coverage—especially for the doc-values path where min/max range checks (lines 71 and 86-88 in Bitmap64DocValuesQuery) must handle negative values correctly.

server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java (1)

76-99: Consider adding edge case tests for the index query path.

The random test is a good fuzz-style check, but explicit tests for edge cases would improve confidence:

• Empty bitmap (verify rewrite to MatchNoDocsQuery)
• Negative long values and values spanning the signed boundary (e.g., -1L, Long.MIN_VALUE)
• equals/hashCode contract verification

The doc-values test file covers some of these for its query type, but the index path has a very different code path (BKD tree intersection) that warrants its own edge case coverage.

server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java (1)

30-30: Cross-class dependency for checkArgs validation.

Bitmap64DocValuesQuery imports checkArgs from Bitmap64IndexQuery. This creates a tight coupling where the doc-values query depends on the index query class for basic validation. This is minor but could be improved by either:

• Extracting checkArgs into a shared utility (e.g., a Bitmap64QueryUtils class), or
• Duplicating the trivial 2-line validation in each class.

📜 Review details

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between df38913 and 0dcc129.

📒 Files selected for processing (5)

• server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java
• server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java
• server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java
• server/src/test/java/org/opensearch/search/query/Bitmap64DocValuesQueryTests.java
• server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java

🧰 Additional context used

🧠 Learnings (1)

📚 Learning: 2026-01-13T17:40:27.167Z

Learnt from: reta
Repo: opensearch-project/OpenSearch PR: 20411
File: server/src/main/java/org/opensearch/index/codec/CodecService.java:112-133
Timestamp: 2026-01-13T17:40:27.167Z
Learning: Avoid capturing or evaluating a supplier (e.g., this::defaultCodec) upfront when passing it to a registry during object construction. If registries may replace defaults during iteration (as in EnginePlugin.getAdditionalCodecs), pass the supplier itself and only resolve it at use time. This ensures dynamic behavior is preserved during initialization and prevents premature binding of defaults in codecs/registry setup. This pattern should apply to similar initialization paths in Java server code where registries may mutate defaults during construction.

Applied to files:

• server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java
• server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java
• server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java

🧬 Code graph analysis (2)

server/src/main/java/org/opensearch/index/mapper/NumberFieldMapper.java (2)

server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java (1)

• Bitmap64IndexQuery (41-244)

server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java (1)

• Bitmap64DocValuesQuery (36-155)

server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java (1)

server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java (1)

• Bitmap64IndexQuery (41-244)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: gradle-check

🔇 Additional comments (7)

server/src/main/java/org/opensearch/search/query/Bitmap64IndexQuery.java (3)

99-139: LGTM — createWeight implementation looks correct.

The weight creates a scorer supplier that uses BKD tree intersection with the merge-sort style MergePointVisitor. The cost heuristic (cardinality × 20) matches the 32-bit version. Returning null when values == null correctly signals no matching docs for the segment. Caching is set to true since the query depends only on the immutable segment point values and the bitmap.

---

210-243: LGTM — rewrite, equals, hashCode, ramBytesUsed look correct.

Empty bitmap rewrites to MatchNoDocsQuery. Equality and hashing delegate to the field and bitmap. RAM usage accounts for the query instance, the field string, and the bitmap's internal memory.

---

62-97: The suggested optimization cannot be applied to Roaring64NavigableMap.

Roaring64NavigableMap#getLongIterator() returns LongIterator, not PeekableLongIterator. The PeekableLongIterator interface with the advanceIfNeeded() method is only available in Roaring64Bitmap, not Roaring64NavigableMap. Applying this suggestion would fail at compile or runtime because the types are incompatible.

To use this optimization, the code would need to either change the input type from Roaring64NavigableMap to Roaring64Bitmap, or use Roaring64Bitmap#getLongIteratorFrom(long minval) which directly accepts a target value. Neither is feasible here without changing the public API of Bitmap64IndexQuery.

Likely an incorrect or invalid review comment.

server/src/test/java/org/opensearch/search/query/Bitmap64DocValuesQueryTests.java (1)

32-219: Good test coverage for the doc-values query path.

The tests cover single-value, multi-value, empty bitmap, rewrite, range, large values, and null-argument validation. This provides solid baseline coverage.

server/src/test/java/org/opensearch/search/query/Bitmap64IndexQueryTests.java (1)

18-138: Overall test structure looks good.

The @Before/@After lifecycle management, the refresh() helper, and the getMatchingValues() extraction utility are well-structured. The tests correctly validate both single-value and multi-value document scenarios.

server/src/main/java/org/opensearch/search/query/Bitmap64DocValuesQuery.java (2)

56-113: createWeight implementation is well-structured.

The singleton vs. multi-value path distinction, the [min, max] range guard before the bitmap.contains() call, and the early termination in the multi-value loop (leveraging SortedNumericDocValues ascending order guarantee) are all correct and efficient. The matchCost() of 5 is a reasonable constant consistent with the 32-bit counterpart.

---

120-126: LGTM — empty bitmap rewrite.

Correctly short-circuits to MatchNoDocsQuery for empty bitmaps, avoiding unnecessary scorer creation.

_{✏️ Tip: You can disable this entire section by setting review_details to false in your review settings.}

[github-actions]

❌ Gradle check result for a0e6ef7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[divyaruhil]

The Gradle Check failure is in

> Task :server:test

Tests with failures:
 - org.opensearch.cluster.routing.remote.RemoteRoutingTableServiceTests.testGetAsyncIndexRoutingTableDiffReadAction

, which appears to be a known Flaky Test 14559

This test passes consistently locally, including with --rerun-tasks.

[divyaruhil]

Hi @andrross @anasalkouz @saratvemulapalli @epugh @kyleconroy I was excited to work on this and have raised the PR. Whenever you get a chance, I’d appreciate a review. I’m happy to update anything based on your suggestions. Thank you!

[github-actions]

✅ Gradle check result for c598688: SUCCESS

[divyaruhil]

Hey @msfroh, I just wanted to take a moment to sincerely thank you for all your support and guidance!

I truly appreciate you taking the time to review my PR, looping in @bowenlan-amzn, and especially for keeping an eye on those flaky Gradle checks — restarting them throughout the day must have taken extra effort, and it really means a lot.

[github-actions]

❌ Gradle check result for 2f917fc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 2f917fc: SUCCESS

[divyaruhil]

Thank you @msfroh, @bowenlan-amzn, and @reta for all your support throughout the review and merge process — I truly appreciate the time and effort you put into it. It means a lot.