Support ppl CAST function with Calcite #3439

LantaoJin · 2025-03-17T10:35:45Z

Description

Support ppl CAST function with Calcite

Related Issues

Resolves #3417

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Lantao Jin <ltjin@amazon.com>

…ssues/3417

LantaoJin · 2025-03-17T10:38:31Z

core/src/main/java/org/opensearch/sql/ast/expression/DataType.java

-  INTERVAL(ExprCoreType.INTERVAL);
+  INTERVAL(ExprCoreType.INTERVAL),
+
+  IP(ExprCoreType.IP);


Cast_to_IP unsupported yet since IP type is not supported with Calcite. But this is a missing code in DataType class.

LantaoJin · 2025-03-17T10:40:15Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

+    RelDataType type = context.rexBuilder.getTypeFactory().createSqlType(sqlTypeName);
+    RelDataType nullableType =
+        context.rexBuilder.getTypeFactory().createTypeWithNullability(type, true);
+    return context.rexBuilder.makeCast(SqlParserPos.ZERO, nullableType, expr, false, true);


use this 4 lines instead of context.relbuilder.cast() because we need the last parameter to true

does Calcite has same Data Type Conversion rule? anything we want to call-out and update doc?

I will add more test for this table

penghuo · 2025-03-17T22:17:59Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLCastFunctionIT.java

+
+  @Test
+  public void testCastToUnknownType() {
+    assertThrows(


what is error message?

penghuo · 2025-03-17T22:20:24Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLCastFunctionIT.java

+                "source=%s | eval a = cast(state as string) | fields a",
+                TEST_INDEX_STATE_COUNTRY_WITH_NULL));
+
+    assertEquals(


why not verifyDataRows?

I don't know how to check null row with verifyDataRows
The method rows() cannot work with null.

penghuo · 2025-03-17T22:23:44Z

integ-test/src/test/java/org/opensearch/sql/calcite/standalone/CalcitePPLCastFunctionIT.java

+    String actual =
+        execute(
+            String.format(
+                "source=%s | eval a = cast(name as boolean) | fields a", TEST_INDEX_STATE_COUNTRY));


cast("1" as boolean) should return True?

no, only 'true', 'True', 'TRUE' etc could return True in cast_to_boolean.
cast(1 as boolean), cast(2.5 as boolean) are all return false and cast("1" as boolean) return null.

This seems a breaking change comparing to V2:
cast("1" as boolean) returns false
cast(1 as boolean), cast(2.5 as boolean) return true
cast(0 as boolean) returns false

Apache Spark:
select cast(1 as boolean); -- true
select cast(2 as boolean); -- true
select cast(0 as boolean); -- false

select cast('1' as boolean); -- true
select cast('2' as boolean); -- null
select cast('0' as boolean); -- false

select cast('aa' as boolean); -- null

Em, I think this is a bug of Calcite, we can fix it in our side.

Open #3443 for discussion

The latest commit follows the behaviour of SparkSQL. Check above

penghuo · 2025-03-17T22:39:15Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

+    RelDataType type = context.rexBuilder.getTypeFactory().createSqlType(sqlTypeName);
+    RelDataType nullableType =
+        context.rexBuilder.getTypeFactory().createTypeWithNullability(type, true);
+    return context.rexBuilder.makeCast(SqlParserPos.ZERO, nullableType, expr, false, true);


does Calcite has same Data Type Conversion rule? anything we want to call-out and update doc?

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin · 2025-03-18T11:19:13Z

does Calcite has same Data Type Conversion rule? anything we want to call-out and update doc?

Call-out:

cast_to_boolean behaviour is different with V2. But we can discuss which would we prefer to, see [BUG] Cast_to_boolean behaviour is different in OpenSearch SQL and Spark SQL #3443
safe-cast behaviour is different with V2. Cast will return null instead of throwing exception if cast fails. But we can discuss which would we prefer to. (Spark/postgre use safe-cast)
cast boolean to float/double trigger a Calcite bug. See the new tests added in CalcitePPLCastFunctionIT

penghuo

Please fix IT before merge.

…ssues/3417

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin · 2025-03-19T06:28:55Z

Failure caused by a flaky test CalcitePPLStringBuiltinFunctionIT > testPosition FAILED, Merging to dev branch.

* init commit Signed-off-by: Lantao Jin <ltjin@amazon.com> * Support ppl CAST function with Calcite Signed-off-by: Lantao Jin <ltjin@amazon.com> * Add more tests Signed-off-by: Lantao Jin <ltjin@amazon.com> * fix ci issue Signed-off-by: Lantao Jin <ltjin@amazon.com> --------- Signed-off-by: Lantao Jin <ltjin@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>

LantaoJin added 3 commits March 17, 2025 17:28

init commit

fcfc2d0

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Support ppl CAST function with Calcite

7df5704

Signed-off-by: Lantao Jin <ltjin@amazon.com>

Merge remote-tracking branch 'upstream/feature/calcite-engine' into i…

b2d39a7

…ssues/3417

LantaoJin added the calcite calcite migration releated label Mar 17, 2025

LantaoJin marked this pull request as ready for review March 17, 2025 10:35

LantaoJin requested review from GumpacG, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners March 17, 2025 10:35

LantaoJin commented Mar 17, 2025

View reviewed changes

penghuo reviewed Mar 17, 2025

View reviewed changes

Add more tests

3dda60c

Signed-off-by: Lantao Jin <ltjin@amazon.com>

penghuo approved these changes Mar 18, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/feature/calcite-engine' into i…

3829d8f

…ssues/3417

LantaoJin mentioned this pull request Mar 19, 2025

[BUG] json function return type is undefined #3336

Closed

fix ci issue

03837de

Signed-off-by: Lantao Jin <ltjin@amazon.com>

LantaoJin merged commit 0022a31 into opensearch-project:feature/calcite-engine Mar 19, 2025
19 of 20 checks passed

LantaoJin mentioned this pull request Mar 19, 2025

[Maintenance] Merge Calcite feature branch to main #3394

Closed

6 tasks

noCharger mentioned this pull request Apr 9, 2025

Add release notes on version 3.0.0.0-beta1 #3529

Merged

2 tasks

LantaoJin mentioned this pull request Apr 11, 2025

[BUG] Cast_to_boolean behaviour is different in OpenSearch SQL and Spark SQL #3443

Closed

	convertedDataType
	: typeName = DATE
	\| typeName = TIME
	\| typeName = TIMESTAMP
	\| typeName = INT
	\| typeName = INTEGER
	\| typeName = DOUBLE
	\| typeName = LONG
	\| typeName = FLOAT
	\| typeName = STRING
	\| typeName = BOOLEAN
	\| typeName = IP
	\| typeName = JSON
	;

Support ppl CAST function with Calcite #3439

Support ppl CAST function with Calcite #3439

Uh oh!

Conversation

LantaoJin commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LantaoJin commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

penghuo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LantaoJin commented Mar 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LantaoJin commented Mar 17, 2025 •

edited

Loading

LantaoJin Mar 18, 2025 •

edited

Loading

LantaoJin Mar 18, 2025 •

edited

Loading

LantaoJin Mar 18, 2025 •

edited

Loading

LantaoJin commented Mar 18, 2025 •

edited

Loading