Skip to content

Conversation

@LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Jan 16, 2025

Description

Build the development framework of Calcite Engine which includes

Parser: Parse existing PPL ANTLR4 grammar to AST
Catalog Binding: Resolve and binding the OpenSearch Indics metadata as schema
Plan Converter: Convert the AST to Calcite logical plan (RelNode)

Related Issues

Resolves #3250

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin LantaoJin added the dev-branch PR is to merge into a dev branch label Jan 16, 2025
Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: Lantao Jin <ltjin@amazon.com>
@LantaoJin
Copy link
Member Author

LantaoJin commented Jan 16, 2025

The IT failed during launching OpenSearch cluster with

[2025-01-16T18:36:12,516][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [integTest-0] fatal error in thread [opensearch[integTest-0][sql-worker][T#1]], exiting
java.lang.ExceptionInInitializerError: null
	at org.apache.calcite.sql2rel.ReflectiveConvertletTable.registerNodeTypeMethod(ReflectiveConvertletTable.java:71) ~[?:?]
	at org.apache.calcite.sql2rel.ReflectiveConvertletTable.<init>(ReflectiveConvertletTable.java:55) ~[?:?]
	at org.apache.calcite.sql2rel.StandardConvertletTable.<init>(StandardConvertletTable.java:121) ~[?:?]
	at org.apache.calcite.sql2rel.StandardConvertletTable.<clinit>(StandardConvertletTable.java:115) ~[?:?]
	at org.apache.calcite.tools.Frameworks$ConfigBuilder.<init>(Frameworks.java:246) ~[?:?]
	at org.apache.calcite.tools.Frameworks$ConfigBuilder.<init>(Frameworks.java:227) ~[?:?]
	at org.apache.calcite.tools.Frameworks.newConfigBuilder(Frameworks.java:211) ~[?:?]
	at org.opensearch.sql.executor.QueryService.buildFrameworkConfig(QueryService.java:142) ~[?:?]
	at org.opensearch.sql.executor.QueryService.execute(QueryService.java:63) ~[?:?]
	at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:65) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) ~[?:?]
	at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) ~[?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:923) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessDeclaredMembers")
	at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:488) ~[?:?]
	at java.base/java.security.AccessController.checkPermission(AccessController.java:1071) ~[?:?]
	at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:411) ~[?:?]
	at java.base/java.lang.Class.checkMemberAccess(Class.java:3227) ~[?:?]
	at java.base/java.lang.Class.getDeclaredMethods(Class.java:2674) ~[?:?]
	at org.apache.calcite.util.ReflectUtil.<clinit>(ReflectUtil.java:63) ~[?:?]
	... 16 more

Even I have set systemProperty 'java.security.manager', 'disallow' or custom a policy file with systemProperty 'java.security.policy', "$policyFile"
the policy file includs

 grant {
    permission java.lang.RuntimePermission "accessDeclaredMembers";
 };

These IT failure could be reproduced in local. Check the log integ-test/build/testclusters/integTest-0/logs/opensearch.stdout.log

@LantaoJin LantaoJin marked this pull request as ready for review January 17, 2025 05:57
@LantaoJin LantaoJin merged commit 4b93d60 into opensearch-project:feature/calcite-engine Jan 17, 2025
6 of 13 checks passed
penghuo pushed a commit to penghuo/os-sql that referenced this pull request Mar 12, 2025
@penghuo penghuo mentioned this pull request Mar 13, 2025
7 tasks
penghuo pushed a commit that referenced this pull request Jun 16, 2025
…er (#3249)

* First commit for Calcite integration

Signed-off-by: Lantao Jin <ltjin@amazon.com>

* disable java security manager in IT

Signed-off-by: Lantao Jin <ltjin@amazon.com>

---------

Signed-off-by: Lantao Jin <ltjin@amazon.com>
Signed-off-by: xinyual <xinyual@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-branch PR is to merge into a dev branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant