`StructArray` field names are not unique

[connortsui20]

This is basically a followup of apache/arrow#27083, which was not migrated to this repository.

I think the defaults for the C++ implementation may have changed since that issue was posted (and I have no idea where the Java implementation has this logic).

• https://github.com/apache/arrow/blob/main/cpp/src/arrow/type.cc#L1419 which calls https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/vector.h#L48. I think that the ability to decide how to deal with conflicts was lost over time...

---

Here is the existing docs: https://docs.rs/arrow/latest/arrow/array/struct.StructArray.html#method.column_by_name. That documentation comment should probably be updated...

In addition to this, I think there are several issues related to StructArray casting and schema evolution that have not taken this behavior into account. Struct casting might be fine because it only looks at the type of the field and not the name? But I can imagine that schema evolution becomes stranger when you can have a bunch of fields with the same name that have different types.

• Support StructArray in Cast Kernel #4908
• Support for "Schema evolution" / Schema Adapters #6735
• Add a way to map RecordBatch schema from one to another #5996

Also see the related discussion in Vortex, as we have a similar behavior now simply because this is what the Rust Arrow implementation does.

[alamb]

Related

• Struct casting requires same order of fields #8870

I am not quite sure what this ticket is proposing (is it proposing to add validation to StructArray creation that all field names are unique?)

FYI @brancz

[connortsui20]

I am not quite sure what this ticket is proposing (is it proposing to add validation to StructArray creation that all field names are unique?)

Yes and no? Yes in that I think the semantics of schema evolution become tricky if you allow duplicates field names, but also no in that if you enforce uniqueness this implementation will differ from the other language impls.

Personally, I don't think it makes a lot of sense to have duplicate fields, but making that change seems to require convincing every arrow language implementation to make a potentially breaking update.

[alamb]

Personally, I don't think it makes a lot of sense to have duplicate fields, but making that change seems to require convincing every arrow language implementation to make a potentially breaking update.

Yes I agree with this assesment

I recall previous versions of this discussion that concluded that Arrow (the spec) didn't impose any field name restrictions, though many downstream consumers often did (e.g. specifically no repeated names)