Add skip_list parameter to Numeric Field Mappers (default false) #18889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

prudhvigodithi merged 8 commits into opensearch-project:main from asimmahmood1:skiplist_mapping

Aug 6, 2025

Contributor

asimmahmood1 commented Aug 1, 2025 •

edited

Loading

Description

add a new boolean parameter skip_list that will enable indexing of skiplists, if doc_values is also true, otherwise it has no effect
this will allow us to experiment with skiplist and decide later when we should enable it by default, and articulate trade off between index, doc_values, and skip_list
TODO: add param to scaled float, size field mapper, token count mapper
BWC: since its a new param, does not affect previous versions

Related Issues

Resolves [#17965]
Replace PR [https://github.com//pull/18066]

Check List

Functionality includes testing.
[] API changes companion pull request TBD, if applicable.
Public documentation issue/PR Add skip_list parameter to numeric field type (default: false) documentation-website#10560

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Contributor

github-actions bot commented Aug 1, 2025

❌ Gradle check result for 2ce6624: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented Aug 1, 2025

❌ Gradle check result for e03141b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

asimmahmood1 force-pushed the skiplist_mapping branch from bef4570 to 0a516d9 Compare

August 1, 2025 19:01


          Add skip_list parameter to Numeric Field Mappers

05f99db

- Add skip_list parameter to numeric field mappers (default false)
- Add mapping unit tests for skip_list functionality

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

asimmahmood1 force-pushed the skiplist_mapping branch from 0a516d9 to 05f99db Compare

August 1, 2025 19:11

Contributor

github-actions bot commented Aug 1, 2025

❌ Gradle check result for 05f99db: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?


          Fix test

ca81788

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

asimmahmood1 added this to Performance Roadmap

github-project-automation bot moved this to Todo in Performance Roadmap

asimmahmood1 moved this from Todo to In Progress in Performance Roadmap

asimmahmood1 added the Performance label

Contributor

github-actions bot commented Aug 2, 2025

Hello!
We have added a performance benchmark workflow that runs by adding a comment on the PR.
Please refer https://github.com/opensearch-project/OpenSearch/blob/main/PERFORMANCE_BENCHMARKS.md on how to run benchmarks on pull requests.

Contributor

github-actions bot commented Aug 2, 2025

❌ Gradle check result for ca81788: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Jul 31, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ResourceAwareTasksTests #14293

Open

[AUTOCUT] Gradle Check Flaky Test Report for TransferManagerBlobContainerReaderTests #18872

Open

asimmahmood1 closed this

github-project-automation bot moved this from In Progress to Done in Performance Roadmap

asimmahmood1 reopened this

github-project-automation bot moved this from Done to In Progress in Performance Roadmap

Contributor

github-actions bot commented Aug 4, 2025

❌ Gradle check result for ca81788: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

This was referenced Aug 4, 2025

[AUTOCUT] Gradle Check Flaky Test Report for TransferManagerRemoteDirectoryReaderTests #16676

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexServiceTests #14407

Open

Contributor Author

asimmahmood1 commented Aug 4, 2025

Going to retry again.

asimmahmood1 closed this

github-project-automation bot moved this from In Progress to Done in Performance Roadmap

asimmahmood1 reopened this

github-project-automation bot moved this from Done to In Progress in Performance Roadmap

Contributor

github-actions bot commented Aug 4, 2025

✅ Gradle check result for ca81788: SUCCESS

asimmahmood1 marked this pull request as ready for review

August 4, 2025 06:15

asimmahmood1 requested a review from anasalkouz as a code owner

August 4, 2025 06:15

asimmahmood1 moved this from In Progress to In-Review in Performance Roadmap


          Merge branch 'main' into skiplist_mapping

7c8a242

Signed-off-by: Asim M <asim.seng@gmail.com>

Contributor

github-actions bot commented Aug 6, 2025

✅ Gradle check result for 7c8a242: SUCCESS

prudhvigodithi merged commit c01ff89 into opensearch-project:main

31 checks passed

github-project-automation bot moved this from In-Review to Done in Performance Roadmap

This was referenced Aug 6, 2025

Update 3.2 changelog to include Numerid field skip_list param #18948

Closed

Update 3.2 changelog to include Numeric field skip_list param #18950

Merged

asimmahmood1 deleted the skiplist_mapping branch

August 6, 2025 22:23

asimmahmood1 mentioned this pull request

Add skip_list parameter to numeric field type (default: false) opensearch-project/documentation-website#10560

Merged

1 task

This was referenced Aug 3, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ClusterRerouteIT #14298

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemotePrimaryLocalRecoveryIT #14314

Open

RajatGupta02 pushed a commit to RajatGupta02/OpenSearch that referenced this pull request


          Add skip_list parameter to Numeric Field Mappers (default false) (ope…

59945fd

…nsearch-project#18889)

* Add skip_list parameter to Numeric Field Mappers

- Add skip_list parameter to numeric field mappers (default false)
- Add mapping unit tests for skip_list functionality

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Fix test

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Remove unused code

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

---------

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
Signed-off-by: Asim M <asim.seng@gmail.com>

asimmahmood1 added this to OpenSearch Roadmap

github-project-automation bot moved this to New in OpenSearch Roadmap

asimmahmood1 added v3.2.0 Search Indexing Indexing & Search Search:Performance labels

Contributor

github-actions bot commented Aug 19, 2025

Hello!
We have added a performance benchmark workflow that runs by adding a comment on the PR.
Please refer https://github.com/opensearch-project/OpenSearch/blob/main/PERFORMANCE_BENCHMARKS.md on how to run benchmarks on pull requests.

asimmahmood1 added the enhancement label

BrewTestBot mentioned this pull request

opensearch 3.2.0 Homebrew/homebrew-core#234146

Merged

asimmahmood1 added a commit to asimmahmood1/OpenSearch that referenced this pull request


          Add skip_list param for scaled float and token count fields

3da39cc

* follow up to opensearch-project#18889, which added to NumberFieldMapper
* remaining are Boolean, which I'm not if its worth adding to
* next step is to enable it by default, pending benchmark

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

This was referenced Aug 25, 2025

Add skip_list param for date, scaled float and token count fields #19142

Merged

Enable skip_list by default in 3.3 #19123

Closed

kh3ra pushed a commit to kh3ra/OpenSearch that referenced this pull request


          Add skip_list parameter to Numeric Field Mappers (default false) (ope…

cf27239

…nsearch-project#18889)

* Add skip_list parameter to Numeric Field Mappers

- Add skip_list parameter to numeric field mappers (default false)
- Add mapping unit tests for skip_list functionality

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Fix test

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Remove unused code

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

---------

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
Signed-off-by: Asim M <asim.seng@gmail.com>

asimmahmood1 added a commit to asimmahmood1/OpenSearch that referenced this pull request


          Add skip_list param for date, scaled float and token count fields

e7b15b4

* follow up to opensearch-project#18889, which added to NumberFieldMapper
* remaining are Boolean, which I'm not if its worth adding to
* next step is to enable it by default, pending benchmark
* update ScaledFloat testRangeQuery to apply same rounding logic to query as indexing to reduce the flakiness

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

asimmahmood1 added a commit to asimmahmood1/OpenSearch that referenced this pull request


          Add skip_list param for date, scaled float and token count fields

183dde7

* follow up to opensearch-project#18889, which added to NumberFieldMapper
* remaining are Boolean, which I'm not if its worth adding to
* next step is to enable it by default, pending benchmark
* update ScaledFloat testRangeQuery to apply same rounding logic to query as indexing to reduce the flakiness

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

asimmahmood1 added a commit to asimmahmood1/OpenSearch that referenced this pull request


          Add skip_list param for date, scaled float and token count fields

c2ab72a

* follow up to opensearch-project#18889, which added to NumberFieldMapper
* remaining are Boolean, which I'm not if its worth adding to
* next step is to enable it by default, pending benchmark
* update ScaledFloat testRangeQuery to apply same rounding logic to query as indexing to reduce the flakiness

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

vinaykpud pushed a commit to vinaykpud/OpenSearch that referenced this pull request


          Add skip_list parameter to Numeric Field Mappers (default false) (ope…

fce787d

…nsearch-project#18889)

* Add skip_list parameter to Numeric Field Mappers

- Add skip_list parameter to numeric field mappers (default false)
- Add mapping unit tests for skip_list functionality

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Fix test

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

* Remove unused code

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>

---------

Signed-off-by: Asim Mahmood <asim.seng@gmail.com>
Signed-off-by: Asim M <asim.seng@gmail.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

bharath-techie bharath-techie approved these changes

anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

andrross Awaiting requested review from andrross andrross is a code owner

ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

cwperks Awaiting requested review from cwperks cwperks is a code owner

dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

jed326 Awaiting requested review from jed326 jed326 is a code owner

kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

mch2 Awaiting requested review from mch2 mch2 is a code owner

msfroh Awaiting requested review from msfroh msfroh is a code owner

owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

reta Awaiting requested review from reta reta is a code owner

Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

sachinpkale Awaiting requested review from sachinpkale sachinpkale is a code owner

saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

sohami Awaiting requested review from sohami sohami is a code owner

VachaShah Awaiting requested review from VachaShah

Labels

enhancement Indexing & Search Indexing Performance Search:Performance Search v3.2.0