Support relevance query functions pushdown implementation in Calcite #3834

songkant-aws · 2025-07-01T03:07:21Z

Description

Support relevance query functions pushdown implementation in Calcite

Related Issues

Resolves #3462

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws · 2025-07-01T03:10:24Z

integ-test/src/test/java/org/opensearch/sql/ppl/MatchPhrasePrefixIT.java

    String query =
        "source = %s| WHERE match_phrase_prefix(Title, 'in to', analyzer=english,"
-            + " zero_terms_query='ALL') | sort -Title | head 1 | fields Title";
+            + " zero_terms_query='ALL') | head 1 | fields Title";


Modify this IT query because sort pushdown is not merged to Calcite yet.

it merged, can you rebase the code?

Rebased the code but the Sort pushdown has different behavior of handling missing_: first/last than V2. Calcite IT still has different assertion. So I leave my change here.

Waiting for @yuancu new PR merging.

Signed-off-by: Songkan Tang <songkant@amazon.com>

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

opensearch/src/main/java/org/opensearch/sql/opensearch/request/PredicateAnalyzer.java

...va/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/MultiFieldQuery.java

...ava/org/opensearch/sql/opensearch/storage/script/filter/lucene/relevance/RelevanceQuery.java

Signed-off-by: Songkan Tang <songkant@amazon.com>

qianheng-aws · 2025-07-01T08:29:14Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

-        if (node.getFuncName().equalsIgnoreCase("reduce")) { // analyze again with calculate type
-          lambdaContext =
+
+    if (SINGLE_FIELD_RELEVANCE_FUNCTION_SET.contains(node.getFuncName())


I don't get why not replying on visitRelevanceFieldList to resolve all fields and then we don't need specific handing for relevance functions.

As discussed, we can support UnresolvedArgument as alias(literal). Making such changes

Addressed UnresolvedArgument change. The amount of changes is less than I expected.

core/src/main/java/org/opensearch/sql/expression/function/PPLBuiltinOperators.java

penghuo · 2025-07-01T17:17:58Z

I don't think we should support relevance query functions directly. The concern is that it would allow users to use them anywhere a predicate expression is valid. However, the query engine does not support relevance evaluation, so it must always push down these functions to the DSL. If pushdown fails, the query will throw an exception.

Instead, I propose supporting relevance-based queries only within the search command.

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws · 2025-07-02T09:11:00Z

@penghuo As discussed, since V2 relevance function cannot be removed quickly. We can mark these functions deprecated once we introduce new syntax of relevance query in search command directly.

opensearch-trigger-bot · 2025-07-04T02:39:07Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3834-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 74b4de067cefc821055076bd108249cef777c3f9
# Push it to GitHub
git push --set-upstream origin backport/backport-3834-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3834-to-2.19-dev.

LantaoJin · 2025-07-04T02:41:38Z

@songkant-aws please manually backport it to 2.19-dev

…pensearch-project#3834) * Support relevance query functions pushdown implementation Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove unused change Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove ITs from NoPushDownIT list Signed-off-by: Songkan Tang <songkant@amazon.com> * Add more UTs and javadocs Signed-off-by: Songkan Tang <songkant@amazon.com> * Support analyzing UnresolvedArgument RexNode and refactor code Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless check Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com>

…entation in Calcite (#3834) (#3848) * Support relevance query functions pushdown implementation in Calcite (#3834) * Support relevance query functions pushdown implementation Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove unused change Signed-off-by: Songkan Tang <songkant@amazon.com> * Remove ITs from NoPushDownIT list Signed-off-by: Songkan Tang <songkant@amazon.com> * Add more UTs and javadocs Signed-off-by: Songkan Tang <songkant@amazon.com> * Support analyzing UnresolvedArgument RexNode and refactor code Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix spotless check Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix compiling issue Signed-off-by: Songkan Tang <songkant@amazon.com> * Fix java syntax Signed-off-by: Songkan Tang <songkant@amazon.com> --------- Signed-off-by: Songkan Tang <songkant@amazon.com>

Support relevance query functions pushdown implementation

634b3c1

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, qianheng-aws, seankao-az and ykmr1224 as code owners July 1, 2025 03:07

Remove unused change

1e84fd3

Signed-off-by: Songkan Tang <songkant@amazon.com>

songkant-aws commented Jul 1, 2025

View reviewed changes

LantaoJin added enhancement New feature or request calcite calcite migration releated pushdown pushdown related issues labels Jul 1, 2025

songkant-aws added 2 commits July 1, 2025 12:41

Remove ITs from NoPushDownIT list

5f718a6

Signed-off-by: Songkan Tang <songkant@amazon.com>

Merge branch 'main' into relevance-query

709f703

LantaoJin reviewed Jul 1, 2025

View reviewed changes

Add more UTs and javadocs

ce1ea5b

Signed-off-by: Songkan Tang <songkant@amazon.com>

qianheng-aws reviewed Jul 1, 2025

View reviewed changes

yuancu reviewed Jul 1, 2025

View reviewed changes

core/src/main/java/org/opensearch/sql/expression/function/PPLBuiltinOperators.java Outdated Show resolved Hide resolved

songkant-aws added 3 commits July 2, 2025 15:15

Support analyzing UnresolvedArgument RexNode and refactor code

18d62a9

Signed-off-by: Songkan Tang <songkant@amazon.com>

Merge branch 'main' into relevance-query

08a6d2b

Fix spotless check

a6215eb

Signed-off-by: Songkan Tang <songkant@amazon.com>

LantaoJin approved these changes Jul 2, 2025

View reviewed changes

LantaoJin mentioned this pull request Jul 2, 2025

[BUG] PPL throws a CalciteUnsupportedException when the query includes both multi_match and json_extract #3771

Closed

qianheng-aws approved these changes Jul 2, 2025

View reviewed changes

qianheng-aws merged commit 74b4de0 into opensearch-project:main Jul 2, 2025
23 checks passed

LantaoJin added the backport 2.19-dev label Jul 4, 2025

opensearch-trigger-bot bot added the backport-failed label Jul 4, 2025

songkant-aws mentioned this pull request Jul 4, 2025

[Backport 2.19-dev] Support relevance query functions pushdown implementation in Calcite (#3834) #3848

Merged

7 tasks

LantaoJin mentioned this pull request Jul 8, 2025

[BugFix] Fix relevance query function over optimization issue in ReduceExpressionsRule #3851

Merged

7 tasks

LantaoJin added the backport-manually Filed a PR to backport manually. label Jul 22, 2025

penghuo mentioned this pull request Jul 30, 2025

Add missing command in index.rst #3943

Merged

7 tasks

songkant-aws deleted the relevance-query branch August 1, 2025 05:27

opensearch-project deleted a comment from opensearch-trigger-bot bot Aug 13, 2025

Support relevance query functions pushdown implementation in Calcite #3834

Support relevance query functions pushdown implementation in Calcite #3834

Uh oh!

Conversation

songkant-aws commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

songkant-aws Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qianheng-aws Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

songkant-aws Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

penghuo commented Jul 1, 2025

Uh oh!

songkant-aws commented Jul 2, 2025

Uh oh!

Uh oh!

opensearch-trigger-bot bot commented Jul 4, 2025

Uh oh!

LantaoJin commented Jul 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

songkant-aws commented Jul 1, 2025 •

edited

Loading