[Umbrella][Feature] Support for complex data types

[XuQianJin-Stars]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Support for complex data types: Array (#168), Map (#169), Struct (#170), Variant/JSON.

Task list

• Support for Row Type Foundation @XuQianJin-Stars
• Support for Array data type @XuQianJin-Stars
• Support for Map data type @XuQianJin-Stars
• Support for Struct data type @XuQianJin-Stars
• Support for Variant/JSON data type

Solution

1. The interface layer will implement: InternalArray, InternalMap, and InternalRow (to be refined).
2. Add generic implementations: GenericArray, GenericMap, and GenericRow (to be refined).
3. Implement ArrayType, MapType, StructType, Variant/JSON, etc., based on Arrow (ColumnarRow), IndexedRow, and CompactedRow respectively.
4. Integrate and transform with Flink/Spark data types.
5. Integrate and transform with data lakes (such as Paimon).

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[georgelza]

Just for my understanding, what we saying here is JSON support, vs the current flat structure only.
to what level might complex include vector based data ?
to what level might complex include bi-modal / binary data (probably to be stored more external) or is this maybe a 0.9.0...

[wuchong]

@georgelza yes, we should add more sub-tasks under this umbrella issue to include the vector and large binary types.

[xx789633]

Hi @XuQianJin-Stars , could you possibly add a task for the BLOB (Binary Large Object) data type? BLOB can enhance our Fluss-Lance connector.