[Feature](avro) Support Apache Avro file format #19990

DongLiang-0 · 2023-05-24T06:06:24Z

Proposed changes

Problem summary

support read avro file by hdfs() or s3() .

select * from s3(
         "uri" = "http://127.0.0.1:9312/test2/person.avro",
         "ACCESS_KEY" = "ak",
         "SECRET_KEY" = "sk",
         "FORMAT" = "avro");
+--------+--------------+-------------+-----------------+
| name   | boolean_type | double_type | long_type       |
+--------+--------------+-------------+-----------------+
| Alyssa |            1 |     10.0012 | 100000000221133 |
| Ben    |            0 |    5555.999 |      4009990000 |
| lisi   |            0 | 5992225.999 |      9099933330 |
+--------+--------------+-------------+-----------------+

select * from hdfs(
                "uri" = "hdfs://127.0.0.1:9000/input/person2.avro",
                "fs.defaultFS" = "hdfs://127.0.0.1:9000",
                "hadoop.username" = "doris",
                "format" = "avro");
+--------+--------------+-------------+-----------+
| name   | boolean_type | double_type | long_type |
+--------+--------------+-------------+-----------+
| Alyssa |            1 |  8888.99999 |  89898989 |
+--------+--------------+-------------+-----------+

current avro reader only support common data type, the complex data types will be supported later.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-05-24T06:12:08Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-06T13:21:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-07T03:37:59Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-12T03:14:34Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-14T05:33:43Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-14T05:50:47Z

clang-tidy review says "All clean, LGTM! 👍"

DongLiang-0 · 2023-06-16T01:46:39Z

run buildall

github-actions · 2023-06-16T01:51:33Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

github-actions · 2023-06-16T03:33:48Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-06-16T03:33:51Z

PR approved by anyone and no changes requested.

DongLiang-0 · 2023-06-16T05:25:40Z

run buildall

AshinGau · 2023-06-16T10:37:31Z

be/src/vec/exec/jni_connector.cpp

@@ -62,6 +62,21 @@ JniConnector::~JniConnector() {
    }
 }

+Status JniConnector::open() {


call open(nullptr, nullptr) directly

Thanks for your suggestion, I have modified here

AshinGau · 2023-06-16T10:48:06Z

be/src/vec/exec/jni_connector.h

+    * Use configuration map to provide scan information. The java side should determine how the parameters are parsed.
+    * @param connector_class Java scanner class
+    * @param scanner_params Provided configuration map
+    */


Add comment to describe the difference between the two constructors.

AshinGau · 2023-06-16T10:49:44Z

be/src/vec/exec/jni_connector.h

+    JniConnector(std::string connector_class, std::map<std::string, std::string> scanner_params)
+            : _connector_class(std::move(connector_class)),
+              _scanner_params(std::move(scanner_params)) {
+        _is_table_schema = true;


I should add identifiers on the Java side and decide how to clean up memory on the Java side.

Java memory is used here, not direct memory, which is managed by JVM

AshinGau · 2023-06-16T10:53:51Z

be/src/vec/exec/jni_connector.h

+     * The schema information are stored as a string.
+     * Use # between column names and column types.
+     *
+     * like: col_name1,col_name2,col_name3#col_type1,col_type2.col_type3


There is an issue with the concatenation method of this string, as type may contain commas, such as the decimal type. You can directly return Map<String, String> type, as shown in the process of passing map types as parameters.

In the support of complex types, I have rebuild this piece. Like array, map types are nested.

AshinGau · 2023-06-16T10:55:00Z

be/src/vec/exec/scan/avro_reader.cpp

+#include "runtime/types.h"
+
+namespace doris::vectorized {
+


Recommendation: AvroReader -> AvroJniReader

Thanks for your suggestion, I have modified here.

AshinGau · 2023-06-16T10:59:49Z

fe/be-java-extensions/avro-scanner/src/main/java/org/apache/doris/avro/AvroColumnValue.java

+
+    @Override
+    public LocalDate getDate() {
+        return LocalDate.now();


LocalDate.now() is only used mock jni reader, please check this.

Maybe it can be changed to return null here？
Avro does not have date type，The code will not run to here either.
what you think？

AshinGau · 2023-06-16T11:01:53Z

fe/be-java-extensions/hudi-scanner/pom.xml

@@ -78,15 +78,16 @@ under the License.
            </exclusions>
        </dependency>
        <dependency>


Why change the hadoop dependencies.

Because there is a conflict between hudi and avro dependencies

AshinGau · 2023-06-16T11:03:04Z

fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/HudiJniScanner.java

@@ -132,6 +133,11 @@ public int getNext() throws IOException {
        }
    }

+    @Override
+    public TableSchema parseTableSchema() throws IOException {


throw UnsupportedOperationException error

AshinGau · 2023-06-16T11:04:13Z

fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/JniScanner.java

@@ -41,6 +42,9 @@ public abstract class JniScanner {
    // Scan data and save as vector table
    protected abstract int getNext() throws IOException;

+    // parse table schema
+    public abstract TableSchema parseTableSchema() throws IOException;


Maybe a protected method.

AshinGau · 2023-06-16T11:11:42Z

...e-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/TableSchema.java

+public class TableSchema {
+
+    private final String[] fields;
+    private final TPrimitiveType[] schemaTypes;


Column type is defined in org.apache.doris.common.jni.vec.ColumnType, maybe we should use ColumnType, not TPrimitiveType. Because ColumnType has provide the parseType method, and get type is an inverse method.

DongLiang-0 · 2023-06-25T05:08:16Z

run buildall

github-actions · 2023-06-25T05:15:05Z

clang-tidy review says "All clean, LGTM! 👍"

DongLiang-0 · 2023-06-25T07:41:13Z

run buildall

DongLiang-0 · 2023-06-25T11:02:57Z

run buildall

github-actions · 2023-06-25T11:07:03Z

clang-tidy review says "All clean, LGTM! 👍"

DongLiang-0 · 2023-06-26T01:24:39Z

run buildall

AshinGau · 2023-06-26T10:46:12Z

fe/be-java-extensions/java-common/src/main/java/org/apache/doris/common/jni/vec/ColumnType.java

@@ -258,6 +258,9 @@ public static ColumnType parseType(String columnName, String hiveType) {
            case "largeint":
                type = Type.LARGEINT;
                break;
+            case "long":


we should generate hive compatible types.

AshinGau · 2023-06-26T11:45:06Z

be/src/vec/exec/jni_connector.h

+     * like: col_name1,col_name2,col_name3#col_type1,col_type2.col_type3
+     *
+     */
+    Status get_table_schema(std::string& table_schema_str);


Map<string, string> get_table_schema();
field_name -> array<struct<col1: int, col2: string>>

Here I use the nested format of json.
You can see TableSchema.SchemaColumn

github-actions · 2023-06-27T01:50:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-06-27T02:01:56Z

clang-tidy review says "All clean, LGTM! 👍"

DongLiang-0 · 2023-06-27T02:06:04Z

run buildall

AshinGau · 2023-06-27T02:45:05Z

LGTM

github-actions · 2023-06-27T02:45:47Z

PR approved by at least one committer and no changes requested.

DongLiang-0 · 2023-06-27T02:46:31Z

run buildall

github-actions · 2023-06-27T02:52:07Z

clang-tidy review says "All clean, LGTM! 👍"

DongLiang-0 · 2023-06-27T07:28:17Z

run buildall

DongLiang-0 · 2023-06-28T01:40:32Z

run buildall

morningman

LGTM

support read avro file by hdfs() or s3() . ```sql select * from s3( "uri" = "http://127.0.0.1:9312/test2/person.avro", "ACCESS_KEY" = "ak", "SECRET_KEY" = "sk", "FORMAT" = "avro"); +--------+--------------+-------------+-----------------+ | name | boolean_type | double_type | long_type | +--------+--------------+-------------+-----------------+ | Alyssa | 1 | 10.0012 | 100000000221133 | | Ben | 0 | 5555.999 | 4009990000 | | lisi | 0 | 5992225.999 | 9099933330 | +--------+--------------+-------------+-----------------+ select * from hdfs( "uri" = "hdfs://127.0.0.1:9000/input/person2.avro", "fs.defaultFS" = "hdfs://127.0.0.1:9000", "hadoop.username" = "doris", "format" = "avro"); +--------+--------------+-------------+-----------+ | name | boolean_type | double_type | long_type | +--------+--------------+-------------+-----------+ | Alyssa | 1 | 8888.99999 | 89898989 | +--------+--------------+-------------+-----------+ ``` current avro reader only support common data type, the complex data types will be supported later.

github-actions bot added area/vectorization kind/docs Categorizes issue or PR as related to documentation. labels May 24, 2023

morningman self-assigned this May 24, 2023

DongLiang-0 force-pushed the avro-format2 branch from 106aa46 to a83a8da Compare June 6, 2023 13:16

DongLiang-0 marked this pull request as draft June 14, 2023 05:32

DongLiang-0 marked this pull request as ready for review June 14, 2023 05:45

morningman previously approved these changes Jun 16, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 16, 2023

github-actions bot added the reviewed label Jun 16, 2023

AshinGau reviewed Jun 16, 2023

View reviewed changes

DongLiang-0 force-pushed the avro-format2 branch from abd1fb8 to 8ba4ae7 Compare June 25, 2023 11:01

AshinGau reviewed Jun 26, 2023

View reviewed changes

DongLiang-0 added 4 commits June 27, 2023 10:24

merge master to solve conflict

75b67c5

fix jni scanner error

6306303

add avro complex type

1810600

fix

a36d011

DongLiang-0 force-pushed the avro-format2 branch from f6b215c to a36d011 Compare June 27, 2023 02:44

AshinGau approved these changes Jun 27, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2023

morningman approved these changes Jun 28, 2023

View reviewed changes

morningman merged commit a6b51ec into apache:master Jun 28, 2023

morningman added the dev/2.0.0-merged label Jul 4, 2023

		#include "runtime/types.h"

		namespace doris::vectorized {

[Feature](avro) Support Apache Avro file format #19990

[Feature](avro) Support Apache Avro file format #19990

Conversation

DongLiang-0 commented May 24, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

github-actions bot commented May 24, 2023

github-actions bot commented Jun 6, 2023

github-actions bot commented Jun 7, 2023

github-actions bot commented Jun 12, 2023

github-actions bot commented Jun 14, 2023

github-actions bot commented Jun 14, 2023

DongLiang-0 commented Jun 16, 2023

github-actions bot commented Jun 16, 2023

morningman left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 16, 2023

github-actions bot commented Jun 16, 2023

DongLiang-0 commented Jun 16, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DongLiang-0 commented Jun 25, 2023

github-actions bot commented Jun 25, 2023

DongLiang-0 commented Jun 25, 2023

DongLiang-0 commented Jun 25, 2023

github-actions bot commented Jun 25, 2023

DongLiang-0 commented Jun 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

DongLiang-0 commented Jun 27, 2023

AshinGau commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

DongLiang-0 commented Jun 27, 2023

github-actions bot commented Jun 27, 2023

DongLiang-0 commented Jun 27, 2023

DongLiang-0 commented Jun 28, 2023

morningman left a comment

Choose a reason for hiding this comment