Support `expand` command with Calcite #3745

yuancu · 2025-06-06T08:25:38Z

Description

Support expand command with Calcite.

Note: expand only explodes nested arrays, not maps.

Related Issues

Resolves #3711

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu · 2025-06-06T08:27:42Z

TODOs:

Test expanding empty arrays
Throw errors when expanding a non-array field (e.g. strings stored in a string field)
Add unit test (challenging since nested arrays are not in place in Calcite's test data)
Update docs (following that of eventstats)

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExpandCommandIT.java

docs/user/ppl/cmd/expand.rst

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

docs/user/ppl/cmd/expand.rst

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

penghuo · 2025-06-09T16:18:55Z

docs/user/ppl/cmd/expand.rst

+============
+     (From 3.1.0)
+
+Use the ``expand`` command on a nested array field to transform a single


nested array -> array? why emphasized nested?

This is because the boundary between array and object is blurry in OpenSearch. For example, a string field can store a string or an array of string. But there isn't an array type of string. Array only exists for nested type, where it stores an array of structs.

The current implementation does not support expanding an array of string stored in a string field since it will only read the string field as string, it doesn't know whether it is an array of string or a single string at the time of generating logical plans. That's why I specifically mentioned nested array.

penghuo · 2025-06-09T16:48:46Z

docs/user/ppl/cmd/expand.rst

+
+expand <field> [as alias]
+
+* field: The field to be expanded (exploded). Currently only nested arrays are


Currently only nested arrays are supported?

It is not limitation of Expand command. Expand command does not aware of OpenSearch nested data type, right?
Is it because PPL engine map OpenSerach nested data type to PPL Array data type?

Yes, expand command does not array of the nested type. This limitation originates from the fact that only nested fields are read as arrays in visitExpand in CalciteRelNodeVisitor. Primitive fields storing arrays are read as primitive types.

penghuo · 2025-06-09T16:56:09Z

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExpandCommandIT.java

+        schema("age", "bigint"),
+        schema("id", "bigint"),
+        schema("address", "struct"));
+    verifyDataRows(


nit: test should be specific, only focus on empty array.

penghuo · 2025-06-09T17:04:28Z

opensearch/src/main/java/org/opensearch/sql/opensearch/executor/OpenSearchExecutionEngine.java

+      // The element type of struct and array is currently set to ANY.
+      // We set them using the runtime type as a workaround.


The element type of struct and array is currently set to ANY.

It is short-term solution, right? if so, add TODO, and issue link?

Yes. Added. Thanks for pointing this out.

penghuo · 2025-06-09T17:16:52Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+    // be used by the right side to correlate with the left side.
+    // Using left join to keep the records where the array field is empty. The corresponding
+    // field in the result will be null.
+    relBuilder.correlate(JoinRelType.LEFT, correlVariable.get().id, List.of(arrayFieldRex));


Create an issue to track enforce limitation on Expand command.

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

…orkaround Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu · 2025-06-10T03:07:25Z

Just to confirm: when the field to expand is an empty array, should I keep it in the result? @LantaoJin @qianheng-aws @penghuo

The current implementation keeps it, with the corresponding expanded value set to null. (This corresponds to left join.)

E.g. for

{"company": "OldEdge", "employees": ["Jack", "Ben"]}
{"company": "NewEdge", "employees": []}

expanding employees results in

{"company": "OldEdge", "employees": "Jack"}
{"company": "OldEdge", "employees": "Ben"}
{"company": "NewEdge", "employees": null}

although technically expanding string arrays is not supported yet

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

* Create scalffolds for implementing expand command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * WIP: Implementing expand command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add array json and index mapping Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * WIP: Implementing expand command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Implement a minimal viable version of expand Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Support expand with alias Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add doc for expand Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * WIP: add unit tests for expand command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Remove unused logical expand & add test cases Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Use left join in expand to keep documents where their expanded array field is empty Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add unit test for expand command Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Update expand doc format Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix: delete test doc with refresh Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Improve expand documentation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Fix typos Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add a version section to expand documentation Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Make expand empty array IT more specific Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Add a limitation section in expand doc & link a issue tracker for a workaround Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Tweak expand command doc Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> * Test expand null field Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> --------- Signed-off-by: Yuanchun Shen <yuanchu@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com>

yuancu added 8 commits June 3, 2025 20:01

Create scalffolds for implementing expand command

112d5ce

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

WIP: Implementing expand command

7220ed0

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Add array json and index mapping

c537b92

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

WIP: Implementing expand command

ae55697

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Implement a minimal viable version of expand

7003abb

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Support expand with alias

9cb2ccb

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Add doc for expand

08b5c43

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Merge main/origin into cmd-expand

b29432d

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

WIP: add unit tests for expand command

10f81b0

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu marked this pull request as ready for review June 6, 2025 12:13

yuancu requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, seankao-az and ykmr1224 as code owners June 6, 2025 12:13

Improve expand documentation

bc7899c

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

LantaoJin previously approved these changes Jun 9, 2025

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExpandCommandIT.java Outdated Show resolved Hide resolved

docs/user/ppl/cmd/expand.rst Outdated Show resolved Hide resolved

docs/user/ppl/cmd/expand.rst Outdated Show resolved Hide resolved

Fix typos

6b264d8

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu dismissed LantaoJin’s stale review via 6b264d8 June 9, 2025 02:30

LantaoJin reviewed Jun 9, 2025

View reviewed changes

docs/user/ppl/cmd/expand.rst Show resolved Hide resolved

Add a version section to expand documentation

fe11b61

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

LantaoJin previously approved these changes Jun 9, 2025

View reviewed changes

LantaoJin added PPL Piped processing language calcite calcite migration releated labels Jun 9, 2025

penghuo reviewed Jun 9, 2025

View reviewed changes

Make expand empty array IT more specific

98fbb12

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu dismissed LantaoJin’s stale review via 98fbb12 June 10, 2025 01:01

Merge origin/main into cmd-expand

f722ccd

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

yuancu mentioned this pull request Jun 10, 2025

[FEATURE] Support creating typed arrays & structs with Calcite #3751

Open

yuancu added 3 commits June 10, 2025 09:26

Add a limitation section in expand doc & link a issue tracker for a w…

5ea6d60

…orkaround Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Merge origin/main into cmd-expand

4c6448d

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Tweak expand command doc

99ad92d

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

Test expand null field

42f5aa3

Signed-off-by: Yuanchun Shen <yuanchu@amazon.com>

penghuo previously approved these changes Jun 10, 2025

View reviewed changes

yuancu dismissed penghuo’s stale review via 42f5aa3 June 10, 2025 04:43

penghuo approved these changes Jun 10, 2025

View reviewed changes

LantaoJin approved these changes Jun 10, 2025

View reviewed changes

yuancu mentioned this pull request Jun 10, 2025

[FEATURE] Enforce memory usage limitation on expand command #3759

Closed

LantaoJin merged commit 939dfe7 into opensearch-project:main Jun 10, 2025
22 checks passed

LantaoJin mentioned this pull request Jun 16, 2025

[FEATURE]New expand_field PPL Command #3016

Closed

LantaoJin mentioned this pull request Jun 23, 2025

#3016 New expand PPL command #3305

Closed

7 tasks

LantaoJin mentioned this pull request Aug 15, 2025

[GitHub Request] Add yuancu as maintainer of OpenSearch SQL opensearch-project/.github#374

Closed

yuancu deleted the cmd-expand branch August 25, 2025 03:01


		expand <field> [as alias]

		* field: The field to be expanded (exploded). Currently only nested arrays are

		// The element type of struct and array is currently set to ANY.
		// We set them using the runtime type as a workaround.

Support expand command with Calcite #3745

Support expand command with Calcite #3745

Uh oh!

Conversation

yuancu commented Jun 6, 2025

Description

Related Issues

Check List

Uh oh!

yuancu commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

penghuo Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yuancu commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Support `expand` command with Calcite #3745

Support `expand` command with Calcite #3745

yuancu commented Jun 6, 2025 •

edited

Loading

penghuo Jun 9, 2025 •

edited

Loading

yuancu commented Jun 10, 2025 •

edited

Loading