Compact ViewArrays on IPCWrite

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The IPC writer typically tries to avoid writing unreferenced data, e.g. re-encoding offsets for StringArray, ListArray, etc... see write_array_data.

Currently the IPC writer writes view data types as is, potentially including significant unreferenced data.

Describe the solution you'd like

I think by default the IPC writer should at least invoke the GC API, and potentially deduplicate them. We could potentially make this configurable.

Describe alternatives you've considered

Additional context

Potentially some overlap with

• Minify Kernel #7186
• Support interleave kernel with deep copy buffers for StringViewArray #7184

FYI @alamb @XiangpengHao

[alamb]

I agree this would be a good feature

[davidhewitt]

I think we've encountered performance trouble for this at Pydantic at service boundaries where we pass IPC streams and will either add a mitigation in our own code or contribute here.

Also, I was wondering if this problem will be relevant for the parquet writer too?

[tustvold]

Also, I was wondering if this problem will be relevant for the parquet writer too?

The parquet writer won't run into this precise issue, although I'm not saying there couldn't be some other ViewArray related issues.

[davidhewitt]

Good to hear, of course I'd expect there may be other things to improve!

[davidhewitt]

I was wondering, does this same problem also apply to dictionaries? From what I can tell from a quick look inside write_array_data, buffers are truncated to any sliced portion but it seems to me that unreferenced dictionary values might still be written (e.g. consider if the dictionary was passed through filter such that only a few keys exist, it looks to me that the values will be passed unchanged through the filter kernel).

I wonder if there's a possible need for a .gc() method on dictionaries which does similar to in the view arrays?

[alamb]

I wonder if there's a possible need for a .gc() method on dictionaries which does similar to in the view arrays?

I do think it would be valuable -- the challenge we have had when working with views in general is deciding when to call gc() / compact as

1. The decision itself can be expensive
2. The operation itself is expensive (as it involves rewriting all the data)

[davidhewitt]

I opened #7683 to discuss.