Support non-optional union types for ORC

[funcheetah]

Summary

Apache Iceberg does not support non-optional union types (e.g. [“int”, “string”]), nor does Apache Spark. This PR enables Iceberg to read non-optional union types by converting them into struct representations for Apache ORC format.

Representation

The struct representations converted from non-option union types are consistent with non-optional union support added in Trino in trinodb/trino#3483.

Deep nested non-optional union types are supported.

Examples

Basic

[“int”, “string”] -> struct<tag int, field0 int, field1 string>

Single type

[“int”] -> int

Related PRs

• Support non-optional union types for Avro #4242

TODO

• Add spec for non-optional union types support

• Handle single type union (e.g. [“int”]) as a primitive type

• Support in non-Spark environments (e.g. iceberg-data, flink, hive, etc.)

• Support for schema pruning within a complex union

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.