ARROW-206: Expose a C++ api to compare ranges of slots between two arrays

[emkornfield]

@wesm the need for this grew out of @fengguangyuan PR to add struct type (#66) and struct builder. I considered a different APIs before settling on this:

1. Add an API that took the parent bitmask (this potentially has possibility of being the most performant, but would have a more awkward contract then provided here)
2. Add an equality comparison for a single slot (leaves the least amount of room for optimization but it would be the simplest to implement).
3. This API which potentially leaves some room for optimization but I think places the least requirements on the caller.

Let me know if you would prefer a different API.

WIP because I need to add more unit tests (I also need to think about if it is worth mirroring the EqualsExact in addition to the Equals method). Which I should get to by the end of the weekend.

@fengguangyuan let me know if this makes sense to you as a way forward on your PR

[fengguangyuan]

@emkornfield
Thanks for your works.
Certainly, these new methods are helpful in some cases, including PR #66, but I have to say that this PR may be a little far away from the topic we discussed in it.

Honestly, but uncertainly however, I've to say, for Struct case, the key point is how to determine the equality of Struct bitmap to the child fields' slots, i.e. whether these bitmaps are comparable. But I don't think any clues are exposed from your current API.

Again, in terms of PR #66, if it's right to compare the slots in Struct bitmap with the corresponding slots in the child fields' bitmap. Or if it's not OK, then, what's the plan to implement the Validate method.
Thank you! :)

[wesm]

const

[wesm]

This seems like a reasonable API to me. I don't think it's worth worrying about performance too much on these methods for the moment until we see applications where it matters.

@fengguangyuan if the struct-level bitmap indicates that the slot is null, there is no need to inspect the child arrays. if the value is not null, then the child arrays' bitmaps and values can be inspected as needed.

[emkornfield]

@fengguangyuan The plan for validate on structs should be that all children arrays are of the same length as the struct array, and that all children arrays are valid. No comparison of null bitmaps.

For equality, (to reiterate Wes's pont), you only check equality for slots on the children that are valid in the parent struct.

[emkornfield]

Are you thinking of the case of start_idx == end_idx or something else?

All else being equal it seems that an empty set should equal an empty. Do you disagree?

[emkornfield]

I had to modify the API to take the start index of the other array, to handle edge cases for list arrays. I added some basic sanity unit tests, but I could add more. I feel not-horrible about the coverage (I think this just end of weekend laziness though), so if you would like me to add more please let me know.

[fengguangyuan]

Thanks for your answering. But currently, the check for equality is ignored, considering its potentially expensive costs.

Personally, I trust your changes will make the API more flexible and stronger. Good work. :)

[wesm]

Looks like a good checkpoint to me. One thought was that the RangeEquals might take the number of elements compared rather than the exclusive end index, but this is fine as is.

+1, thank you

[xhochy]

Just rebased my current changes on this and realised a disadvantage of it: PrimtiveArray is now an abstract class, not sure if we want to keep this.

[emkornfield]

@xhochy can you give more details on where this caused you problems?

[xhochy]

I used to instantiate a PrimitiveArray directly here: https://github.com/apache/arrow/pull/83/files#diff-89c0372c3577568ee3310523d6f05defR178 Using MakePrimitiveArray is definitely the better way after some thought.

[emkornfield]

How does this sound as a resolution (I can open up the JIRA once we agree)?

1. Make PrimitiveArray's constructor protected
2. Update the documentation on the class to point users to MakePrimitiveArray.

[xhochy]

Sounds good!