[Paimon Catalog] Querying some types of data from paimon raise an error

[0akarma]

After several tests, I can summarize if row columns include BOOLEAN/TIMESTAMP/nested struct(row) types in the table it will raise the same error.

Steps to reproduce the behavior (Required)

-- DROP TABLE row_error_test;
CREATE TABLE IF NOT EXISTS row_error_test (
    row_field ROW<
        f_sequence INT,
        f_random INT,
        f_random_str STRING,
        f_random_boolean BOOLEAN,
        f_random_timestamp TIMESTAMP,
        nest_row_field ROW<
            f_random_boolean BOOLEAN
        >
    >
);
-- DROP TEMPORARY TABLE datagen_source;
CREATE TEMPORARY TABLE datagen_source (
    row_field ROW<
        f_sequence INT,
        f_random INT,
        f_random_str STRING,
        f_random_boolean BOOLEAN,
        f_random_timestamp TIMESTAMP,
        nest_row_field ROW<
            f_random_boolean BOOLEAN
        >
    >
) WITH (
 'connector' = 'datagen',
 'number-of-rows' = '10',
 'fields.row_field.f_sequence.min'='1',
 'fields.row_field.f_sequence.max'='1000',

 'fields.row_field.f_random.min'='1',
 'fields.row_field.f_random.max'='1000',

 'fields.row_field.f_random_str.length'='100'
);


INSERT INTO row_error_test select * from datagen_source;

Expected behavior (Required)

Flink SQL return resut below

Flink SQL> select *, row_field.f_random_boolean, row_field.f_random_timestamp, row_field.nest_row_field from row_error_test limit 1;
2023-09-08 22:15:27,675 INFO  org.apache.hadoop.fs.s3a.S3AInputStream                      [] - Switching to Random IO seek policy
2023-09-08 22:15:27,694 INFO  org.apache.hadoop.fs.s3a.S3AInputStream                      [] - Switching to Random IO seek policy
2023-09-08 22:15:27,713 INFO  org.apache.hadoop.fs.s3a.S3AInputStream                      [] - Switching to Random IO seek policy
+--------------------------------+------------------+----------------------------+----------------+
|                      row_field | f_random_boolean |         f_random_timestamp | nest_row_field |
+--------------------------------+------------------+----------------------------+----------------+
| (1, 951, 4cc6d7b60565f01f82... |            FALSE | 2023-09-08 14:11:12.285000 |         (TRUE) |
+--------------------------------+------------------+----------------------------+----------------+

Real behavior (Required)

Sql-client return an error below

StarRocks > select *, row_field.f_random_boolean, row_field.f_random_timestamp, row_field.nest_row_field from row_error_test limit 1;
ERROR 1064 (HY000): Failed to call the nextChunkOffHeap method of off-heap table scanner. java exception details: java.io.IOException: Failed to get the next off-heap table chunk of paimon.[com.starrocks.paimon.reader.PaimonSplitScanner.getNext(PaimonSplitScanner.java:215), com.starrocks.jni.connector.ConnectorScanner.getNextOffHeapChunk(ConnectorScanner.java:101)]

StarRocks version (Required)

• You can get the StarRocks version by executing SQL select current_version()

in branch main and the latest commit is da50131

[0akarma]

Hi, @leoyy0316 I think It is related to the support for the struct type in this PR(#30162)

[leoyy0316]

fixed by #30817

[0akarma]

LGTM

[0akarma]

Hi, @leoyy0316. I found other uncovered cases like these

-- works fine, row colum dump as json string
StarRocks > select * from row_error_test limit 1;
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| row_field                                                                                                                                                                                                                                                      |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| {"f_sequence":98,"f_random":438,"f_random_str":"e0d13196a632b2a2bf621f508d55f3e62d134458d440fb9ee8a6f71fb5267c3177b1a20a5c040b02d3c4e6cbe11efd649489","f_random_boolean":1,"f_random_timestamp":"2023-09-14 04:30:01","nest_row_field":{"f_random_boolean":1}} |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.45 sec)

-- work fine, int type in row cast correctly
StarRocks > select row_field.f_sequence from row_error_test limit 1;
+------------+
| f_sequence |
+------------+
|         98 |
+------------+
1 row in set (0.34 sec)

-- raise error when casting string / timestamp / nested row type
StarRocks > select row_field.f_random_str from row_error_test limit 1;
ERROR 1064 (HY000): Failed to call the nextChunkOffHeap method of off-heap table scanner. java exception details: java.io.IOException: Failed to get the next off-heap table chunk of paimon.
        at com.starrocks.paimon.reader.PaimonSplitScanner.getNext(PaimonSplitScanner.java:215)
        at com.starrocks.jni.connector.ConnectorScanner.getNextOffHeapChunk(ConnectorScanner.java:101)
Caused by: java.lang.ClassCastException

-- be error when casting nested row type
Caused by: java.lang.ClassCastException: class org.apache.paimon.format.orc.reader.OrcLongColumnVector cannot be cast to class org.apache.paimon.data.columnar.RowColumnVector (org.apache.paimon.format.orc.reader.OrcLongColumnVector and org.apache.paimon.data.columnar.RowColumnVector are in unnamed module of loader com.starrocks.utils.loader.ChildFirstClassLoader @778f8a65)
        at org.apache.paimon.data.columnar.VectorizedColumnBatch.getRow(VectorizedColumnBatch.java:128)
        at org.apache.paimon.data.columnar.ColumnarRow.getRow(ColumnarRow.java:150)
        at org.apache.paimon.utils.InternalRowUtils.get(InternalRowUtils.java:192)
        at com.starrocks.paimon.reader.PaimonColumnValue.unpackStruct(PaimonColumnValue.java:132)
        at com.starrocks.jni.connector.OffHeapColumnVector.appendValue(OffHeapColumnVector.java:498)
        at com.starrocks.jni.connector.OffHeapTable.appendData(OffHeapTable.java:95)
        at com.starrocks.jni.connector.ConnectorScanner.appendData(ConnectorScanner.java:86)
        at com.starrocks.paimon.reader.PaimonSplitScanner.getNext(PaimonSplitScanner.java:205)
        ... 1 more