Variant Data Type Support

[sfc-gh-aixu]

Proposed Change

We would like to propose to add Variant type to Iceberg data types.

Variant data types allow for the efficient binary encoding of dynamic semi-structured data such as JSON, Avro,Parquet, etc. By encoding semi-structured data as a variant column, we retain the flexibility of the source data, while allowing query engines to more efficiently operate on the data.

With the support of Variant type, such data can be encoded in an efficient binary representation internally for better performance. Without that, we need to parse the data in its format inefficiently.

This will allow the following use cases:

• Create an Iceberg table with a Variant column
  CREATE OR REPLACE TABLE car_sales(record Variant);
• Insert semi-structured data into the Variant column
  INSERT INTO car_sales SELECT PARSE_JSON(<json_string>)
• Query against the semi-structured data
  SELECT VARIANT_GET(record, '$.dealer.ship', 'string') FROM car_sales

Proposal document

https://docs.google.com/document/d/1sq70XDiWJ2DemWyA5dVB80gKzwi0CWoM0LOWM7VJVd8/edit?tab=t.0

Specifications

• Table
• View
• REST
• Puffin
• Encryption
• Other

[soumilshah1995]

+1

[aihazm]

+1

[soumilshah1995]

Any ideas on release dates ?

[NurFaizin]

+1

[tmnd1991]

+1

[soumilshah1995]

Any updates on release date for Varient Feature ?

[aihuaxu]

Hi folks. We are actively working on the Variant support. I have made a POC with Spark implementation and currently we are working on individual PRs to move forward. The full Variant support includes part1 - basic Variant type encoding/decoding (without shredding) and part2 - Variant with shredding. We will get part1 to work soon. The spec for Part2 is getting finalized and we will work on the design.