[EPIC] [Parquet] Implement Variant type support in Parquet

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Parquet recently adopted the Variant type from Spark: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

Details on

• GH-455: Add Variant specification docs parquet-format#456

Describe the solution you'd like
I would like to implement variant support in parquet-rs

Additional context
I am not sure if any other parquet implementations have implemented this yet / if there are example parquet files. I will attempt to find out

Related Tasks

• Support round tripping extension types in parquet #7063
• Add example Variant data and parquet files parquet-testing#75
• Variant: Rust API to Read Variant Values #7423
• Variant: Rust API to Create Variant Values #7424
• Variant: Read/Parse JSON value as Variant #7425
• Variant: Write Variant Values as JSON #7426
• Add example Variant data and parquet files parquet-testing#75

Related PRs

• [PATHFINDING] Parse json as variant #7403
• Variant Support for Arrow and Parquet [DRAFT] #7404

** Related Community Resources**

• Parquet C/C++ variant implementation from @neilechao: GH-45937: [C++][Parquet] Variant logical type definition arrow#45375
• Rust Variant impelementation from @jonhoo @wjones127 and others: https://github.com/datafusion-contrib/datafusion-functions-variant
• [ ]

[CurtHagenlocher]

There's an implementation in Spark (try here for starters) but when I last looked ~two months ago there was no binary test data; only some round trips via JSON.

[tustvold]

I do wonder if a precursor to supporting this would be some way to translate / represent the variant data in arrow, whilst there are non-arrow APIs, they'd likely struggle to accommodate this addition, and aren't how the vast majority of people consume parquet data using this crate.

[findepi]

From arrow perspective, would that be a new DataType, or rather a convention of using DataType::Struct with two Binary fields?

A fully performance variant implementation should be able to leverage file-level column disaggregation (shredding), but I do think this could come as a follow-up to a "normal" Variant type implementation.

[tustvold]

From arrow perspective, would that be a new DataType, or rather a convention of using DataType::Struct with two Binary fields?

I don't know, I've not really been following the variant proposal close enough to weigh in here. However, my understanding is that shredding is one of the major motivators for this getting added to parquet, as without it you might as well just embed any record format, e.g. Avro. I therefore suspect most use-cases will be at least partially shredded, and the reader will need to handle this case. This is especially true given the variant_value is NULL when the data is shredded, as opposed to say duplicating the content (which would have its own issues TBC), and so we can't just ignore the shredded data.

Unfortunately I can't see an obvious way to be able to represent this sort of semi-structured data within the arrow format without introducing a new DataType that is able to accommodate arrays having the same type, but different child layouts...

TLDR I suspect actioning this will require arrow defining a way to represent semi-structured data...

[findepi]

There needs to be a way to represent a series of variant values having "no type in common" (variant integer, variant boolean, variant varchar, etc all mixed up). For that some blob-like representation with internal structure seems natural.
Then there should be a way to carry-on the shredded columns without having to put them back into that blob, so yes, one type, different child layouts.
It feels to me that the runtime representation will end up being similar to what is defined in Parquet (https://github.com/apache/parquet-format/blob/master/VariantShredding.md)... so maybe it should be the same representation to provide for an efficient read path.

[findepi]

When considering what to do in Arrow, we should also keep an eye on the ongoing effort in Iceberg apache/iceberg#10392 (comment)
This could inform some design decisions.
cc @Xuanwo

[alamb]

There is now a related proposal being added to paruqet (shredding the variant type:

• Simplify Variant shredding and refactor for clarity parquet-format#461

FWIW I would love to have some rust representation (aka someone who wanted to implement the variant type in the rust parquet decoder). If you are interested there are relevant conversations going on on the parquet mailing list

[alamb]

to be clear, we may not be able to merge / really use variants in rust until it gets into Arrow, but we can work out how it would work in parquet (maybe with the non arrow interfaces) first

[CurtHagenlocher]

There's a pure Python implementation in the Spark repo. It's almost standalone, having a dependency only on PySparkValueError.