Add new APIs to AddressableNonNullValueList to copy/append a stream of bytes

[aditi-pandit]

This is the first in a set of PRs to add support for spilling distinct aggregations (see full version in #7791).

Spilling distinct aggregations needs support to spill SetAccumulators in which input values are cumulated. ComplexTypeSetAccumulators use AddressableNonNullValueList to serialize complex types. This PR adds new APIs to AddressableNonNullValueList so that it can copy/append a stream of bytes corresponding to a ComplexType value (array, map, struct).

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	a3b0469
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/65c2c0a5eb18fa000867fd5e

[mbasmanova]

@aditi-pandit Aditi, can you remind me how are you planning to serialize distinct accumulators? Will you extract each accumulator into a single blob (VARBINARY values) or an array of blobs (ARRAY(VARBINARY)) or something else?

Would it be more efficient to serialize / deserialize all entries in bulk rather than one at a time?

[aditi-pandit]

@mbasmanova : Answers inline

@aditi-pandit Aditi, can you remind me how are you planning to serialize distinct accumulators? Will you extract each accumulator into a single blob (VARBINARY values) or an array of blobs (ARRAY(VARBINARY)) or something else?

The implementation extracts each accumulator into a single blob. #8660 has the SetAccumulator serialization code.

Would it be more efficient to serialize / deserialize all entries in bulk rather than one at a time?

Do you mean serialize multiple ComplexTypes in one go ? Or do you mean something else ?

One issue for serializing multiple ComplexTypes is that we need a hash of each ComplexType value at a time for ComplexTypeSetAccumulator (using BaseVector::hashValueAt() logic). If there is a way to reconstruct such a hash from the ComplexType value one at a time, then doing a bulk write of values and reconstructing hash on read is possible. Right now, I didn't see a way to do this in the code. But its possible to do some refactoring for it.

wdyt ?

[aditi-pandit]

@mbasmanova : Answers inline

@aditi-pandit Aditi, can you remind me how are you planning to serialize distinct accumulators? Will you extract each accumulator into a single blob (VARBINARY values) or an array of blobs (ARRAY(VARBINARY)) or something else?

The implementation extracts each accumulator into a single blob. #8660 has the SetAccumulator serialization code.

Would it be more efficient to serialize / deserialize all entries in bulk rather than one at a time?

Do you mean serialize multiple ComplexTypes in one go ? Or do you mean something else ?

One issue for serializing multiple ComplexTypes is that we need a hash of each ComplexType value at a time for ComplexTypeSetAccumulator (using BaseVector::hashValueAt() logic). If there is a way to reconstruct such a hash from the ComplexType value one at a time, then doing a bulk write of values and reconstructing hash on read is possible. Right now, I didn't see a way to do this in the code. But its possible to do some refactoring for it.

wdyt ?

Confirmed with Masha about this limitation for bulk serialize/deserialize.

[aditi-pandit]

@mbasmanova : PTAL. Thanks.

[mbasmanova]

@aditi-pandit Overall looks good % some comments.

[aditi-pandit]

@mbasmanova : PTAL.

[aditi-pandit]

@mbasmanova : Thanks for your review comments. PTAL.

[mbasmanova]

Thanks.

[facebook-github-bot]

@mbasmanova has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[aditi-pandit]

@mbasmanova : There are failures in Facebook Linter for this. Do you need any help from me ? Would be great to get the error messages.

[facebook-github-bot]

@mbasmanova merged this pull request in 02a4f6c.

[conbench-facebook]

Conbench analyzed the 1 benchmark run on commit 02a4f6c3.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.