Skip to content

Commit ca3b3be

Browse files
authored
[Variant] Define and use unshred_variant function (#8481)
# Which issue does this PR close? - Closes #8336 - Closes #8334 # Rationale for this change The `VariantArray::value` method was really inefficient but had quietly crept into important code paths like `variant_get`. Since the complex and inefficient code was in support of variant unshredding, we should just add and use a proper `unshred_variant` function (which uses row builders like the other variant manipulating functions). # What changes are included in this PR? * Define the new `unshred_variant` function, which does what it says. It supports all the types `typed_value_to_variant` supported, plus Time64 and Struct as a bonus. The former because it was ~10LoC and the latter because it demonstrates the superiority of this new approach vs. e.g. #8446 * Wire up `variant_get` unshredding path to call it, which immediately benefits from all that function's existing test coverage. * Update the variant_integration test to `unshred_variant` instead of looping over rows calling `value(i)`. # Are these changes tested? Yes, a bunch of variant integration tests now pass that used to fail. # Are there any user-facing changes? Several new pub methods. I don't think any breaking changes.
1 parent b9c2bf7 commit ca3b3be

File tree

4 files changed

+586
-45
lines changed

4 files changed

+586
-45
lines changed

parquet-variant-compute/Cargo.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ indexmap = "2.10.0"
3838
parquet-variant = { workspace = true }
3939
parquet-variant-json = { workspace = true }
4040
chrono = { workspace = true }
41+
uuid = { version = "1.18.0", features = ["v4"]}
4142

4243
[lib]
4344
name = "parquet_variant_compute"

parquet-variant-compute/src/lib.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ mod from_json;
4141
mod shred_variant;
4242
mod to_json;
4343
mod type_conversion;
44+
mod unshred_variant;
4445
mod variant_array;
4546
mod variant_array_builder;
4647
pub mod variant_get;
@@ -54,3 +55,4 @@ pub use from_json::json_to_variant;
5455
pub use shred_variant::shred_variant;
5556
pub use to_json::variant_to_json;
5657
pub use type_conversion::CastOptions;
58+
pub use unshred_variant::unshred_variant;

0 commit comments

Comments
 (0)