[Feature] Stream load support JSON to STRUCT/MAP/ARRAY #45406

rickif · 2024-05-10T04:03:44Z

Why I'm doing:

Now, the SR does't support loading JSON to complex type column.

What I'm doing:

This PR adds the support of JSON format loading to STRUCT/MAP/ARRAY type column.

Here are some examples.

create table tbl_complex (col_int int, col_struct struct<key1 int>, col_array array<int>, col_map map<string,int>, col_struct_array struct<key3 array<int>>,
col_struct_map struct<key4 map<string,int>>);


curl --location-trusted -u root:  'http://127.0.0.1:18040/api/db0/tbl_complex/_stream_load' \-X PUT  \-H 'format: json' -d '{"col_int":1,"col_struct":{"key1":1}, "col_map":{"key2":2}, "col_array":[1,2,3], "col_struct_array":{"key3":[2,3,4,5]}, "col_struct_map":{"key4": {"key5":4}}}'

This feature also works for routine load.

Fixes #43101

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

rickif · 2024-05-12T06:53:31Z

@Mergifyio rebase

mergify · 2024-05-12T06:53:56Z

rebase

✅ Branch has been successfully rebased

be/src/formats/json/nullable_column.cpp

be/src/formats/json/struct_column.cpp

be/src/formats/json/map_column.cpp

jaogoy · 2024-05-21T02:37:45Z

be/src/exec/json_scanner.cpp

@@ -194,6 +194,12 @@ Status JsonScanner::_construct_json_types() {
            break;
        }

+        case TYPE_STRUCT:


Does it support nested Array with struct or map as it's subfield data type?
It seems that it will treat all other types as varchar in TYPE_ARRAY in above code.

Updated. The nested types are supported now.

jaogoy · 2024-05-21T03:22:35Z

be/src/formats/json/struct_column.cpp

+
+            auto field_column = struct_column->field_column(field_name);
+            simdjson::ondemand::value field_value = obj.find_field_unordered(field_name);
+            RETURN_IF_ERROR(add_nullable_column(field_column.get(), field_type_desc, name, &field_value, true));


name OR field_name ?
What is the name for?

The name is the column name. It's used to report error.

jaogoy · 2024-05-21T03:27:51Z

be/src/formats/json/map_column.cpp

+            {
+                // This is a tricky way to transform a std::string to simdjson:ondemand:value
+                std::string_view field_name_str = field.unescaped_key();
+                auto dummy_json = simdjson::padded_string(R"({"dummy_key": ")" + std::string(field_name_str) + R"("})");


What is the tricy way? Does the field_key need to be a JSON value rather than a simple string, so you parse a dummy_json to get the key? Why not implemet a add_nullable_column with string value?

Yeah. Since the last argument ofadd_nullable_column is simdjson::ondemand::value, we have to convert the simple string to simdjson::ondemand::value.
There're many functions using the simdjson::ondemand::value as the input argument, implementing a add_nullable_column with string/float/double/int value will involve massive changes.

jaogoy · 2024-05-21T03:29:14Z

be/src/formats/json/map_column.cpp

+    auto map_column = down_cast<MapColumn*>(column);
+
+    try {
+        simdjson::ondemand::object obj = value->get_object();


if value is not an object, what will be the error message?

An exception The JSON element does not have the requested type. will be thrown and be caught later.

Can we give more detailed info, such as the expected type is xxx, the real given type is yyy. It will be much clearer.

jaogoy · 2024-05-21T03:33:41Z

be/src/formats/json/map_column.cpp

+
+namespace starrocks {
+
+Status add_map_column(Column* column, const TypeDescriptor& type_desc, const std::string& name,


How about extracting array processing code to add_array_column?

jaogoy · 2024-05-21T03:56:20Z

be/test/formats/json/nullable_column_test.cpp

@@ -111,4 +112,34 @@ TEST_F(AddNullableColumnTest, add_null_numeric_array) {
    column->check_or_die();
 }

+TEST_F(AddNullableColumnTest, test_add_struct) {


How about nested Struct/map/array?

Yes, they are supported. I've added more information in the PR description.

meegoo · 2024-05-21T08:54:11Z

be/src/formats/json/struct_column.cpp

+            const auto& field_name = type_desc.field_names[i];
+            const auto& field_type_desc = type_desc.children[i];
+
+            auto field_column = struct_column->field_column(field_name);


is it case sensitive?

Yes. If the struct field name has a different case than the JSON field name, the struct field value will be a NULL.

rickif · 2024-05-28T23:59:21Z

@Mergifyio rebase

Signed-off-by: ricky <rickif@qq.com>

Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: ricky <rickif@qq.com>

github-actions · 2024-05-29T04:10:17Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-05-29T04:20:57Z

[BE Incremental Coverage Report]

✅ pass : 118 / 134 (88.06%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/formats/json/nullable_column.cpp	38	46	82.61%	[213, 214, 234, 235, 249, 250, 267, 268]
🔵	be/src/formats/json/struct_column.cpp	16	19	84.21%	[29, 30, 31]
🔵	be/src/formats/json/map_column.cpp	25	28	89.29%	[30, 31, 32]
🔵	be/src/exec/json_scanner.cpp	39	41	95.12%	[279, 550]

wyb · 2024-05-30T05:16:05Z

https://github.com/Mergifyio backport branch-3.3

mergify · 2024-05-30T05:16:18Z

backport branch-3.3

✅ Backports have been created

#46448 [Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406) has been created for branch branch-3.3

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> (cherry picked from commit a4f53d3)

…) (#46448) Co-authored-by: ricky <rickif@qq.com>

wyb · 2024-06-20T01:28:34Z

https://github.com/Mergifyio backport branch-3.2

mergify · 2024-06-20T01:28:53Z

backport branch-3.2

✅ Backports have been created

#47233 [Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406) has been created for branch branch-3.2

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> (cherry picked from commit a4f53d3)

…) (#47233) Co-authored-by: ricky <rickif@qq.com>

rickif requested review from a team as code owners May 10, 2024 04:03

mergify bot assigned rickif May 10, 2024

rickif force-pushed the feat/json-to-struct branch from c50c31d to 6fe62fc Compare May 11, 2024 12:04

rickif requested a review from a team as a code owner May 11, 2024 12:04

rickif force-pushed the feat/json-to-struct branch 6 times, most recently from 08a04c8 to 5153047 Compare May 13, 2024 06:34

github-actions bot added the behavior_changed label May 13, 2024

wyb reviewed May 20, 2024

View reviewed changes

be/src/formats/json/nullable_column.cpp Show resolved Hide resolved

be/src/formats/json/nullable_column.cpp Outdated Show resolved Hide resolved

be/src/formats/json/struct_column.cpp Show resolved Hide resolved

be/src/formats/json/map_column.cpp Outdated Show resolved Hide resolved

jaogoy reviewed May 21, 2024

View reviewed changes

meegoo reviewed May 21, 2024

View reviewed changes

rickif force-pushed the feat/json-to-struct branch from e5169ac to eeb8056 Compare May 28, 2024 14:15

rickif and others added 11 commits May 28, 2024 23:59

*: support json to struct/map

81c5d83

Signed-off-by: ricky <rickif@qq.com>

format: clang-format

8cc4afb

Signed-off-by: ricky <rickif@qq.com>

test: add AddStructColumnTest

ac346bf

Signed-off-by: ricky <rickif@qq.com>

test: add AddMapColumnTest

10e5711

Signed-off-by: ricky <rickif@qq.com>

map_column : map size

33ed0c2

Signed-off-by: ricky <rickif@qq.com>

test: AddMapColumnTest

91e489c

Signed-off-by: ricky <rickif@qq.com>

test: update JsonScannerTest

fef0502

Signed-off-by: ricky <rickif@qq.com>

test: update AddNullableColumnTest

321ea76

Signed-off-by: ricky <rickif@qq.com>

*: clean code

332f05b

Signed-off-by: ricky <rickif@qq.com>

Update be/src/formats/json/nullable_column.cpp

110aefa

Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: ricky <rickif@qq.com>

Update be/src/formats/json/map_column.cpp

9c5481b

Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: ricky <rickif@qq.com>

rickif dismissed jaogoy’s stale review via 30adfb6 May 29, 2024 02:32

meegoo enabled auto-merge (squash) May 29, 2024 02:44

meegoo approved these changes May 29, 2024

View reviewed changes

wyb approved these changes May 29, 2024

View reviewed changes

trueeyu approved these changes May 29, 2024

View reviewed changes

dirtysalt approved these changes May 29, 2024

View reviewed changes

meegoo merged commit a4f53d3 into StarRocks:main May 29, 2024
61 checks passed

github-actions bot added the version:3.4 label May 29, 2024

wanpengfei-git mentioned this pull request May 29, 2024

Support converting JSON to STRUCT #43101

Open

3 tasks

mergify bot pushed a commit that referenced this pull request May 30, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (#45406)

3a18269

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> (cherry picked from commit a4f53d3)

mergify bot mentioned this pull request May 30, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406) #46448

Merged

42 tasks

wanpengfei-git pushed a commit that referenced this pull request Jun 1, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406…

4701738

…) (#46448) Co-authored-by: ricky <rickif@qq.com>

github-actions bot added the 3.3-merged label Jun 1, 2024

wyb mentioned this pull request Jun 13, 2024

[BugFix] Fix routine load json char type #46469

Merged

24 tasks

StarRocks deleted a comment from mergify bot Jun 20, 2024

mergify bot pushed a commit that referenced this pull request Jun 20, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (#45406)

9451a64

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> (cherry picked from commit a4f53d3)

mergify bot mentioned this pull request Jun 20, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406) #47233

Merged

42 tasks

wanpengfei-git pushed a commit that referenced this pull request Jun 20, 2024

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY (backport #45406…

07a78b8

…) (#47233) Co-authored-by: ricky <rickif@qq.com>

github-actions bot added the 3.2-merged label Jun 20, 2024

mergify bot mentioned this pull request Jun 20, 2024

[BugFix] Fix routine load json char type (backport #46469) #47236

Merged

42 tasks

This was referenced Jul 3, 2024

[BugFix] Fill null if struct field is not found in json #47457

Merged

[BugFix] Bump simdjson to 3.9.4 and Fix struct field columns inconsistent when loading from bad json #47775

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY #45406

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY #45406

rickif commented May 10, 2024 •

edited by wyb

Loading

rickif commented May 12, 2024

mergify bot commented May 12, 2024

jaogoy May 21, 2024

rickif May 28, 2024

jaogoy May 21, 2024

rickif May 21, 2024

jaogoy May 21, 2024

rickif May 21, 2024

jaogoy May 21, 2024

rickif May 21, 2024

jaogoy May 29, 2024

jaogoy May 21, 2024

jaogoy May 21, 2024

rickif May 28, 2024

meegoo May 21, 2024

rickif May 22, 2024

rickif commented May 28, 2024

github-actions bot commented May 29, 2024

github-actions bot commented May 29, 2024

wyb commented May 30, 2024

mergify bot commented May 30, 2024 •

edited

Loading

wyb commented Jun 20, 2024

mergify bot commented Jun 20, 2024 •

edited

Loading


		namespace starrocks {

		Status add_map_column(Column* column, const TypeDescriptor& type_desc, const std::string& name,

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY #45406

[Feature] Stream load support JSON to STRUCT/MAP/ARRAY #45406

Conversation

rickif commented May 10, 2024 • edited by wyb Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

rickif commented May 12, 2024

mergify bot commented May 12, 2024

✅ Branch has been successfully rebased

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rickif commented May 28, 2024

github-actions bot commented May 29, 2024

[FE Incremental Coverage Report]

github-actions bot commented May 29, 2024

[BE Incremental Coverage Report]

file detail

wyb commented May 30, 2024

mergify bot commented May 30, 2024 • edited Loading

✅ Backports have been created

wyb commented Jun 20, 2024

mergify bot commented Jun 20, 2024 • edited Loading

✅ Backports have been created

rickif commented May 10, 2024 •

edited by wyb

Loading

mergify bot commented May 30, 2024 •

edited

Loading

mergify bot commented Jun 20, 2024 •

edited

Loading