[Feature] Add native format writer to access StarRocks data bypass BE Server. #52700

plotor · 2024-11-07T09:41:07Z

Why I'm doing:

You can learn about the background from PR #52682.

What I'm doing:

In order to support bypassing the BE server, and directly reading and writing the data stored in the object storage in share-data mode, we extracted a format-sdk module. This SDK:

Provides the C++ version of Reader and Writer implementation, which can read and write StarRocks data files.
Provides the Java version of Reader and Writer (invokes the C++ Reader and Writer through JNI), which can be used by computing engines.
Implements data exchange through the Arrow format, making it easier for more programming languages to access.

This PR mainly contains the implementation of Writer. You can build it by:

mvn clean package -DskipTests

This will complete the compilation of C++ and Java codes, and finally generate the corresponding so and jar files. Then, you can call the SDK to perform data writing operations as follows (taking Java as an example):

// Create and open the Writer
StarRocksWriter srWriter = new StarRocksWriter(...);
srWriter.open();

// Write data, which organized in Arrow format
VectorSchemaRoot vsr = VectorSchemaRoot.create(schema, writer.getRootAllocator());
srWriter.write(vsr);

// Close the Writer
srWriter.close();

Fixes: #52682

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
This is a backport pr

Bugfix cherry-pick branch check:

decster · 2024-12-12T02:58:30Z

be/CMakeLists.txt

@@ -968,9 +968,15 @@ endif()

 # Add all external dependencies. They should come after the starrocks libs.

+set(STARROCKS_STD_DEPENDENCIES
+        -static-libstdc++


why add this? these will be add later

if (NOT ("${MAKE_TEST}" STREQUAL "ON" AND "${BUILD_FOR_SANITIZE}" STREQUAL "ON")) # In other words, turn to dynamic link when MAKE_TEST and BUILD_TYPE == *SAN # otherwise do static link gcc's lib set(STARROCKS_LINK_LIBS ${STARROCKS_LINK_LIBS} -static-libstdc++ -static-libgcc) endif()

OK, I have removed it

decster · 2024-12-12T03:40:05Z

format-sdk/README.md

@@ -0,0 +1 @@
+# StarRocks Format Library


add some information?

As planned, I will write a comprehensive document here in the next PR, including the use of Writer and Reader.

decster · 2024-12-13T15:29:41Z

format-sdk/src/main/cpp/convert/CMakeLists.txt

+# specific language governing permissions and limitations
+# under the License.
+
+FILE(GLOB CONVERTER_SRC_FILES "*.cpp" "*.h")


I recall GLOB will affect incremental build, better list files explicitly

Good advice, learned a lot.

decster · 2024-12-13T15:38:15Z

format-sdk/src/main/cpp/convert/column_converter.h

+    constexpr int32_t ARROW_CONVERTER_ID(arrow::Type::type arrow_type_id, starrocks::LogicalType sr_logical_type) {
+        return (arrow_type_id << 16) | (sr_logical_type);


Add some assert/DCHECK to check overflow

decster · 2024-12-13T15:54:19Z

format-sdk/src/main/cpp/convert/column_converter.cpp

+    std::shared_ptr<starrocks::Column> ColumnConverter::get_data_column(
+            const std::shared_ptr<starrocks::Column> &column) {
+        if (column->is_nullable()) {
+            auto *nullable_column = down_cast<const NullableColumn *>(column.get());
+            return nullable_column->data_column();
+        } else if (column->is_constant()) {
+            auto *const_column = down_cast<const ConstColumn *>(column.get());
+            return const_column->data_column();
+        } else {
+            return column;
+        }
+    }


looks like a static method?

decster · 2024-12-13T15:59:37Z

format-sdk/src/main/cpp/convert/column_converter.cpp

+        ARROW_ASSIGN_OR_RAISE(
+                null_bitmap, arrow::internal::BytesToBits(
+                reinterpret_cast<const std::vector<uint8_t> &>(null_bytes),
+                const_cast<arrow::MemoryPool *>(_pool))


why make _pool const, then use const_cast?

To make _pool as const is mainly to prevent it from being changed in the column converter, but the param pool received by the BytesToBits method is a non-const type, so a const_cast conversion is done as a last resort.

decster · 2024-12-13T16:03:47Z

format-sdk/src/main/cpp/format/starrocks_format_writer.h

+        virtual arrow::Status open() = 0;
+
+        virtual arrow::Status write(const struct ArrowArray *c_arrow_array) = 0;
+
+        virtual arrow::Status flush() = 0;
+
+        virtual arrow::Status finish() = 0;
+
+        virtual void close() = 0;


Add comment document about how this class should be used, open/write/flush/finish order.

OK, good suggention

decster · 2024-12-13T16:10:15Z

format-sdk/src/main/cpp/jni_wrapper.cpp

+            return JNI_ERR;
+        }
+
+        kNativeOptExceptionClass = find_class(env, "Lcom/starrocks/format/jni/NativeOperateException;");


I thought FindClass does not need Lxxx; in "Lcom/starrocks/format/jni/NativeOperateException;" , should you double confirm this?

I checked the Oracle JNI docs, and the Lxxx here is not needed. I have removed it.

… Server. Signed-off-by: plotor <zhenchao.wang@hotmail.com>

sonarqubecloud · 2024-12-16T13:04:04Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2024-12-16T15:00:46Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-12-16T15:03:30Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2024-12-16T15:11:07Z

[BE Incremental Coverage Report]

✅ pass : 14 / 15 (93.33%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/column/field.h	2	3	66.67%	[113]
🔵	be/src/types/timestamp_value.cpp	5	5	100.00%	[]
🔵	be/src/types/date_value.cpp	5	5	100.00%	[]
🔵	be/src/exec/olap_scan_prepare.cpp	2	2	100.00%	[]

mergify bot assigned plotor Nov 7, 2024

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch 4 times, most recently from 300bdd1 to 1c009e6 Compare November 7, 2024 11:19

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch 4 times, most recently from 127475c to 69e0a59 Compare November 27, 2024 08:50

plotor marked this pull request as ready for review November 27, 2024 09:44

plotor requested a review from a team as a code owner November 27, 2024 09:44

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch from 69e0a59 to d6f19f3 Compare November 27, 2024 09:46

plotor mentioned this pull request Nov 27, 2024

[Feature] Supports reading and writing data files bypass BE Server #52682

Closed

8 tasks

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch 5 times, most recently from e662ef3 to 45d87f8 Compare December 5, 2024 08:48

decster reviewed Dec 13, 2024

View reviewed changes

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch 4 times, most recently from d1b9dec to 9e6aa3a Compare December 16, 2024 11:43

[Feature] Add native format writer to access StarRocks data bypass BE…

388ef6d

… Server. Signed-off-by: plotor <zhenchao.wang@hotmail.com>

plotor force-pushed the zhenchao-bypass-format-writer-20241107 branch from 9e6aa3a to 388ef6d Compare December 16, 2024 12:57

stdpain approved these changes Dec 18, 2024

View reviewed changes

decster approved these changes Dec 18, 2024

View reviewed changes

decster merged commit ed87f90 into StarRocks:main Dec 18, 2024
51 of 52 checks passed

github-actions bot added the version:3.5 label Dec 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add native format writer to access StarRocks data bypass BE Server. #52700

[Feature] Add native format writer to access StarRocks data bypass BE Server. #52700

plotor commented Nov 7, 2024 •

edited by decster

Loading

decster Dec 12, 2024

plotor Dec 16, 2024

decster Dec 12, 2024

plotor Dec 16, 2024

decster Dec 13, 2024

plotor Dec 16, 2024

decster Dec 13, 2024

decster Dec 13, 2024

decster Dec 13, 2024

plotor Dec 16, 2024

decster Dec 13, 2024

plotor Dec 16, 2024

decster Dec 13, 2024

plotor Dec 16, 2024

sonarqubecloud bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

github-actions bot commented Dec 16, 2024

		constexpr int32_t ARROW_CONVERTER_ID(arrow::Type::type arrow_type_id, starrocks::LogicalType sr_logical_type) {
		return (arrow_type_id << 16) \| (sr_logical_type);

[Feature] Add native format writer to access StarRocks data bypass BE Server. #52700

[Feature] Add native format writer to access StarRocks data bypass BE Server. #52700

Conversation

plotor commented Nov 7, 2024 • edited by decster Loading

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 16, 2024

Quality Gate passed

github-actions bot commented Dec 16, 2024

[Java-Extensions Incremental Coverage Report]

github-actions bot commented Dec 16, 2024

[FE Incremental Coverage Report]

github-actions bot commented Dec 16, 2024

[BE Incremental Coverage Report]

file detail

plotor commented Nov 7, 2024 •

edited by decster

Loading