GH-38007: [C++] Add VariableShapeTensor implementation

[rok]

Rationale for this change

We want to add VariableShapeTensor extension type definition for arrays containing tensors with variable shapes.

What changes are included in this PR?

This adds a C++ implementation.

Are these changes tested?

Yes.

Are there any user-facing changes?

This adds a new extension type C++.

• Closes: [C++][Python] Add VariableShapeTensor implementation #38007
• GitHub Issue: [C++][Python] Add VariableShapeTensor implementation #38007

[AlenkaF]

Hi @rok! It would be great to have this in 15.0.0 (added the milestone for visibility). Do you think you have enough bandwidth to work on it at the moment?

[rok]

@AlenkaF yes I'd very much like to continue working on this, I was waiting for the release before chasing reviewers :D
I'll rebase tonight and would very much appreciate reviews afterwards.

[rok]

As I understand it, the FixedShapeTensorArray has less need for a Large version, because the FixedSizeListType does not hold an offset int32 array like the ListType does. So 1 array can hold many tensors, as long as each is less than 2^31 elements. The VariableShapeTensorArray as it's currently documented can hold at most 2^31 elements in all tensors in the array combined, so that limit is hit earlier. Also, there doesn't seem to be a LargeFixedSizeListType, which has an int64 size?

@pimdh Yes, that seems right.
Now that some time has passed would you say there's still need for VariableShapeTensorArray with LargeList for data? We would need to amend the extension type or introduce LargeVariableShapeTensorArray.

[rok]

Given that there is some interest in using VariableShapeTensor and that it was already implemented in rust it might make sense to merge this PR. @pitrou would you happen to have time to take another look here?

[pimdh]

@pimdh Yes, that seems right.
Now that some time has passed would you say there's still need for VariableShapeTensorArray with LargeList for data? We would need to amend the extension type or introduce LargeVariableShapeTensorArray.

No, as I didn't yet run into any limits of the int32 arrays, I have no need for the large version.
Appreciate your work on this! Would love to replace our internal version with an official one.
Also, then others, like the Ray library, can use the standardised version.