Support full expression in WHERE clauses

[LantaoJin]

Description

PPL where clause throws syntax error for:

• SOURCE=test | WHERE x
• SOURCE=test | WHERE x OR y
• SOURCE=test | WHERE true
• SOURCE=test | WHERE (x < 1) = (y > 1)

Related Issues

Resolves #3273, #3272 and #3317

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[LantaoJin]

Ping @qianheng-aws @dai-chen @penghuo @Swiddis

[LantaoJin]

The SEARCH keyword changed to required since the query describe source=t could be ambiguous.
describe source=t should throw syntax error "describe source= <==== .."
But it could be matched to

(SEARCH)? logicalExpression fromClause

logicalExpression -> valueExpr -> fieldExpr -> describe
fromClause -> source=t

[LantaoJin]

Not sure why we supported the syntax of (SEARCH)? logicalExpression fromClause.

[penghuo]

search keywords is optional, this is valid query status=200 source=index

[LantaoJin]

updated, now the logicalExpression could be both before and after source clause which is similar to SPL search command.

searchCommand
  : (SEARCH)? (logicalExpression)* fromClause (logicalExpression)*     # searchFrom

[LantaoJin]

The AND will be required in Where expression. But it is still optional in Search filter expression:

source=t a=1 b=c

✅

source=t | where a=1 b=c

❌ (SPL cannot work either)

source=t | where a=1 and b=c

✅

This limitation is because functionArgs could be ambiguous. For example:

source=t | eval f=position('substr' IN 'str')

can be parsed to an incorrect syntax tree:

The correct syntax tree is

[qianheng-aws]

LGTM if above 2 limitation are acceptable.

Another concern is that we may not able to push down such filter condition of single field in DSL. But that issue could be addressed in a separate PR.

[LantaoJin]

@penghuo this pr did some refactoring on antlr g4 file, could you take another look? There are multiple PPL new command PRs filed, to avoid conflicts, let's complete this refactoring ASAP.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.x
# Create a new branch
git switch --create backport/backport-3849-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 988ab2ea13d533f761558c18c0844c4f206a18d6
# Push it to GitHub
git push --set-upstream origin backport/backport-3849-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-3849-to-2.x.

[opensearch-trigger-bot]

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-3849-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 988ab2ea13d533f761558c18c0844c4f206a18d6
# Push it to GitHub
git push --set-upstream origin backport/backport-3849-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-3849-to-2.19-dev.