Variant: Rust API to Read Variant Values

[alamb]

• Part of [EPIC] [Parquet] Implement Variant type support in Parquet #6736

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The first part of supporting the Variant type in Parquet and Arrow is
programmatic access to values encoded with the binary format described in
[VariantEncoding.md]. This ticket covers the API to read such values, but not
creating such values, or representing it using arrow or parquet which are
covered in other tickets

Describe the solution you'd like
I would like a Rust API, similar to the Json::Value and similar APIs to dynamically access variant values.

Here is some example binary data for testing:

• Add example binary variant data and regeneration scripts parquet-testing#76

Describe alternatives you've considered

I think a Rust enum approach with references would be a good model.

I suggest creating a new crate, arrow-variant, and marking it as
experimental, etc saying it will contain breaking changes for the next several
releases (maybe we can even version it 0.1, etc)

For example:

Sketch of structures

/// Variant value. May contain references to metadata and value
/// 'a is lifetime for metadata
/// 'b is lifetime for value
pub enum Variant<'a, 'b> {
  Variant::Null,
  Variant::Int8
  ...
  // strings are stored in the value and thus have references to that value
  Variant::String(&'b str),
  Variant::ShortString(&'b str),
  // Objects and Arrays need the metadata and values, so store both.
  Variant::Object(VariantObject<'a, 'b>),
  VariantArray(VariantArray<'a, 'b>)
}

/// Wrapper over Variant Metadata
pub struct VariantMetadata<'a> {
  metadata: &'a[u8],
  // perhaps access to header fields like dict length and is_sorted
}

/// Represents a Variant Object with references to the underlying metadata
/// and value fields
pub enum VariantObject<'a, 'b> {
  // pointer to metadata
  metadata: VariantMetadata<'a>,
  // pointer to value
  value: &'a [u8],
}

Creating Variants from buffers

// Each variant has a metadata and value buffer:
let metadata: &[u8] = ...;
let value: &[u8] = ....;
// The Rust API should NOT require allocations or copy the metadata/values
let variant = Variant::try_new(metadata, value)?;

Working with Primitive Variants

// Act based on the type of variant
match variant {
  Variant::Int8(val) => println!("The value was int8: {val}"),
  ...
  Variant::SmallString(val) => println!("The value was a small string: {val}"),
  ...
  Variant::Object(object) => {
    println("The variant was in object. The fields are:");
    for (field_name, field_value) in object.fields()? {
      // The inner field value is also a variant
      match field_value {
        Variant::...
      }
    }
  }
  // similarly for Variant::Array
}

I personally suggest doing this over a few PRs:

1. Scaffolding: Variant struct/enum, support a few basic variant primtive types
2. Basic nested type support: basic support for objects
3. Array support: support for arrays
4. Complete APIs, etc

Additional context

• @PinkCrow007s: Variant Support for Arrow and Parquet [DRAFT] #7404. https://github.com/apache/arrow-rs/blob/0220e97b407cf6690ac8b51d53fa0e92273f1d8c/arrow-schema/src/extension/canonical/variant.rs#L38-L37
• Spark implementation: https://github.com/apache/spark/blob/007c31df6da7b741f3a2a43c859ebed9f801dcfa/common/variant/src/main/java/org/apache/spark/types/variant/Variant.java#L43
• Python implementation (also in spark): https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py
• Examples from @jonhoo and @wjones127 in https://github.com/datafusion-contrib/datafusion-functions-variant

Open Questions:When should validation be done?

I do think there should be an API like:

/// ensure that metadata and value are valid according to the Variant spec, returns error if not.
Variant::validate(&metadata, &value)?;

However, the API sketched above proposes doing validation on access (when the
values are accessed). An alternate approach would be to validate everything on
creation and then use unchecked APIs during access.

I think validating once upfront is better if most fields are accessed or certain
fields are read multiple times. For the usecase where only some fields are read
I think verifying on access would be faster.

The spec also allows metadata to contain dictionary values that do not appear as
struct names in the variant value itself, so eager validation would potentially
verify string data uncessairly.

I suggest starting with an API that is fallible (aka creating a Variant or
accessing a field returns Result<Variant>. We can always add unsafe versions
of the APIs for usecases where validation overhead is significant (e.g. writing
utf8 validation for field names when writing json), and justified with benchmarks

[mkarbo]

We'd like to pick this up, can you assign this one to me @alamb?

[alamb]

Thanks @mkarbo !

[alamb]

FYI @PinkCrow007

[alamb]

BTW there are a few known issues with the example variant values apache/parquet-testing#75 in parquet-testing:

Specifically

• Potential issues with Null variant example parquet-testing#81
• primitive_int64.value maybe an int32 type parquet-testing#82

I think they will be relatively easy to solve / workaround for the time being, but I wanted to bring them to your attention

[alamb]

@scovich had some great comments on #7452 (comment) that I wanted to copy/paste into this ticket perhaps for wider discussion:

I have not reviewed the code carefully at all yet, and what follows is a general observation based on the inherent nature of variant data and rust notions of safety:

It will be really tempting to have "efficient" code that e.g. uses from_utf8_unchecked to extract a &str from a &[u8], or to use indexing operations like v[10] to extract bytes. But variant data is generally untrusted user input and whatever Variant struct/enum we define will become the first -- and often only -- line of defense against malicious or malformed input.

Hopefully we can code carefully, with the goal that sizes and/or contents of metadata and value slices will never cause a panic?

Additinoally, it seems like we have a few choices for values such as strings and decimals even a right-sized byte slice can contain invalid values:

1. Return obviously unvalidated values, e.g. &[u8] instead of &str for strings, and &[u8] instead of whatever VariantDecimal struct we might otherwise define -- leaving the user responsible to finish the conversion as (un)safely as they deem prudent.
2. Return ostensibly validated values, with (safe) checked and (unsafe) unchecked constructors and/or getters that let the user choose the one they deem appropriate.

I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion.

[alamb]

I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion.

I likewise think the "safe / return error by default" is the right model. I also agree it should be an API goal that there should be no panics on invalid variant data (it should return an error instead)

[mkarbo]

Yeah I agree, thanks for pointing it out