[Calcite Engine]Support metadata field #3445

qianheng-aws · 2025-03-19T06:42:30Z

Description

Support metadata field in Calcite. This PR also include changes:

Fix the bug on v2 engine that cannot evaluate metadata field if it's not pushdown
Add limitation of using metadata field on eval, alias and rename

Related Issues

Resolves #3333

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws · 2025-03-20T06:12:12Z

flakey failure:
CalcitePPLStringBuiltinFunctionPushdownIT > testSubstring

Signed-off-by: Heng Qian <qianheng@amazon.com>

…calcite-engine-metadata # Conflicts: # core/src/main/java/org/opensearch/sql/ast/expression/Alias.java # integ-test/src/test/java/org/opensearch/sql/calcite/remote/nonfallback/NonFallbackCalciteFieldsCommandIT.java # ppl/src/main/java/org/opensearch/sql/ppl/parser/AstStatementBuilder.java

Signed-off-by: Heng Qian <qianheng@amazon.com>

…calcite-engine-metadata # Conflicts: # integ-test/src/test/java/org/opensearch/sql/calcite/remote/nonfallback/NonFallbackCalciteFieldsCommandIT.java # integ-test/src/test/java/org/opensearch/sql/calcite/remote/nonfallback/NonFallbackCalciteWhereCommandIT.java

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin

Please describe the limitation in https://github.com/opensearch-project/sql/blob/main/docs/dev/intro-v3-engine.md#33-limitations

LantaoJin · 2025-03-20T15:08:43Z

core/src/main/java/org/opensearch/sql/ast/expression/Alias.java

+    this(name, expr, alias, false);
+  }
+
+  public Alias(String name, UnresolvedExpression expr, boolean metaMetaFieldAllowed) {


Why double “meta”?

Alias is a public interface. Could you add some comments in code to illustrate the purpose of this parameter.

Will make it private and add comments

LantaoJin · 2025-03-21T03:47:55Z

core/src/main/java/org/opensearch/sql/ast/expression/AllFields.java

    return nodeVisitor.visitAllFields(this, context);
  }
+
+  public UnresolvedPlan apply(UnresolvedPlan plan) {


can you add a java doc for this interface? can this method be moved to CalcitePlanContext and rename to wrapProject?

Doc added. We cannot do that because some of this wrapping happens in AstBuilder

LantaoJin · 2025-03-21T03:53:53Z

core/src/main/java/org/opensearch/sql/ast/tree/Rename.java

+    checkRename();
+  }
+
+  private void checkRename() {


validate()

LantaoJin · 2025-04-16T09:22:50Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLRenameIT.java

+                executeQuery(
+                    String.format(
+                        """
+                   source = %s | rename name as _id


what if rename name as _ID or rename name as _Id? if it is valid, could you add a test for these valid new names

Added another test case for it.

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin · 2025-04-17T06:49:36Z

core/src/main/java/org/opensearch/sql/ast/expression/AllFields.java

+  public static final AllFields INSTANCE_OF_ALL = new AllFields(false);
+  public static final AllFields INSTANCE_EXCEPT_META = new AllFields(true);


Can we create a new derived class or classes for different semantics?

Signed-off-by: Heng Qian <qianheng@amazon.com>

penghuo · 2025-04-21T19:39:40Z

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java

-      return StringUtils.format("[ %s ] as %s", child, node.getAlias());
+    Node childNode = node.getChild().get(0);
+    String child = childNode.accept(this, context);
+    if (childNode instanceof Project project


add an UT to cover this?

It's covered in PPLQueryDataAnonymizerTest::testJoin, there are several test cases with subquery alias

penghuo · 2025-04-21T22:38:29Z

core/src/main/java/org/opensearch/sql/storage/Table.java

  }

+  /** Whether include reserved fields in the schema of table */
+  default boolean includeReservedFieldTypes() {


not been used?

Yeah, will delete it.

integ-test/src/test/resources/expectedOutput/calcite/explain_filter_push.json

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin · 2025-04-22T13:54:25Z

Found a bug after merged this PR:
source=[source=banks | where age>35] as t throws NPE when Calcite disabled.

* [Calcite Engine]Support metadata field Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix UT/IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> * [Calcite engine] Fix nested subquery Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix merging Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine Code Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments3 Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix Anonymizer Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> (cherry picked from commit 9092cc8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [Calcite Engine]Support metadata field * Fix UT/IT * Refine code * [Calcite engine] Fix nested subquery * Fix merging * Refine Code * Fix IT * Address comments * Address comments * Address comments3 * Address comments * Fix Anonymizer * Address comments --------- (cherry picked from commit 9092cc8) Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* [Calcite Engine]Support metadata field Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix UT/IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine code Signed-off-by: Heng Qian <qianheng@amazon.com> * [Calcite engine] Fix nested subquery Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix merging Signed-off-by: Heng Qian <qianheng@amazon.com> * Refine Code Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix IT Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments3 Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> * Fix Anonymizer Signed-off-by: Heng Qian <qianheng@amazon.com> * Address comments Signed-off-by: Heng Qian <qianheng@amazon.com> --------- Signed-off-by: Heng Qian <qianheng@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>

[Calcite Engine]Support metadata field

5960440

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners March 19, 2025 06:42

qianheng-aws marked this pull request as draft March 19, 2025 14:04

Fix UT/IT

f86021f

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws marked this pull request as ready for review March 20, 2025 05:24

Refine code

6c5fc81

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin added the calcite calcite migration releated label Mar 20, 2025

[Calcite engine] Fix nested subquery

6b18f9d

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws changed the base branch from feature/calcite-engine to main March 25, 2025 02:39

qianheng-aws added 2 commits March 25, 2025 10:51

Fix merging

6674503

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws added 3 commits April 15, 2025 16:48

Refine Code

43349b0

Signed-off-by: Heng Qian <qianheng@amazon.com>

Fix IT

f3828f7

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin reviewed Apr 16, 2025

View reviewed changes

qianheng-aws added 2 commits April 17, 2025 12:16

Address comments

8107574

Signed-off-by: Heng Qian <qianheng@amazon.com>

Address comments

1f29c87

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws requested a review from LantaoJin April 17, 2025 05:50

LantaoJin reviewed Apr 17, 2025

View reviewed changes

qianheng-aws added 3 commits April 17, 2025 15:23

Address comments3

e01d44e

Signed-off-by: Heng Qian <qianheng@amazon.com>

Address comments

c4f83ae

Signed-off-by: Heng Qian <qianheng@amazon.com>

Fix Anonymizer

7ea78be

Signed-off-by: Heng Qian <qianheng@amazon.com>

LantaoJin previously approved these changes Apr 17, 2025

View reviewed changes

penghuo reviewed Apr 21, 2025

View reviewed changes

integ-test/src/test/resources/expectedOutput/calcite/explain_filter_push.json Show resolved Hide resolved

Address comments

275ec03

Signed-off-by: Heng Qian <qianheng@amazon.com>

qianheng-aws dismissed LantaoJin’s stale review via 275ec03 April 22, 2025 03:21

penghuo approved these changes Apr 22, 2025

View reviewed changes

qianheng-aws requested a review from LantaoJin April 22, 2025 05:37

LantaoJin approved these changes Apr 22, 2025

View reviewed changes

LantaoJin merged commit 9092cc8 into opensearch-project:main Apr 22, 2025
22 checks passed

LantaoJin mentioned this pull request Apr 22, 2025

New added commands should throw exception when calcite disabled #3571

Merged

7 tasks

LantaoJin added bug Something isn't working backport 3.0 labels Apr 25, 2025

opensearch-trigger-bot bot mentioned this pull request Apr 25, 2025

[Backport 3.0] [Calcite Engine]Support metadata field #3581

Merged

vamsimanohar mentioned this pull request Sep 18, 2025

Fix geopoint issue in complex data types #4325

Merged

8 tasks

		public static final AllFields INSTANCE_OF_ALL = new AllFields(false);
		public static final AllFields INSTANCE_EXCEPT_META = new AllFields(true);

[Calcite Engine]Support metadata field #3445

[Calcite Engine]Support metadata field #3445

Uh oh!

Conversation

qianheng-aws commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

qianheng-aws commented Mar 20, 2025

Uh oh!

LantaoJin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Apr 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LantaoJin commented Apr 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qianheng-aws commented Mar 19, 2025 •

edited

Loading

qianheng-aws Apr 22, 2025 •

edited

Loading