improved support for struct columns with missing values

[baumgold]

The current implementation depends on Arrow.default returning a sentinel object when the value is missing. This usually works, but occasionally has problems. This PR simplifies the implementation and solves some of the problematic corner-cases.

One example of a problematic corner-case is a column of lists of structs:

julia> [(a=1,b=2), missing, (a=3,b=4)]
3-element Vector{Union{Missing, @NamedTuple{a::Int64, b::Int64}}}:
 (a = 1, b = 2)
 missing
 (a = 3, b = 4)

It's possible that users may wrap this Vector in a SubArray. In this case Arrow.default(::Type{<:SubArray }) actually uses Arrow.default(::Type{<:AbstractVector}) which enforces that the parent-type is a Vector - this is not always the case. For example, the parent may well be of type Arrow.Struct if the data being written was also read from an Arrow file. Certainly we could enhance Arrow.default to return a proper type for SubArray but simpler is to remove the dependency on Arrow.default from ToStruct.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (ac199b0) 87.34% compared to head (78e22be) 87.36%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #498      +/-   ##
==========================================
+ Coverage   87.34%   87.36%   +0.01%     
==========================================
  Files          26       26              
  Lines        3288     3292       +4     
==========================================
+ Hits         2872     2876       +4     
  Misses        416      416              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[baumgold]

@quinnj / @ericphanson - Any questions/comments/concerns here? If not I'd like to merge and release. Thanks.

[ericphanson]

do we still want Base.nonmissingtype(eltype(A)) inside the field type?

[baumgold]

I don't think so. My change makes it so missing types are properly supported.

[ericphanson]

I don't really know this code well enough to be confident here. But one thing I can say is it's probably good if the PR adds a test that is failing on main, to show precisely what has been fixed (and prevent regressions)