Avro version conflict between Hive and Iceberg libraries

[lxynov]

Currently Hive depends on Avro 1.8.2 whereas Iceberg depends on 1.9.2.

In the Iceberg Connector, we exclude Avro dependency from iceberg-core, so Iceberg has been run with Avro 1.8.2 in Presto. It turns out to be fine for Iceberg 0.9.0 because it doesn't seem to be using any class not available in Avro 1.8.2. However, a new version Iceberg 0.10.0-rc0 breaks because it uses a class InvalidAvroMagicException that is not available in Avro 1.8.2. As a result, this Avro version conflict becomes a blocker for us to bump Iceberg version.

To resolve this, we can consider:

1. Bump Hive's Avro version to 1.9.2.
2. Bump both Hive's Avro version and Iceberg's to 1.10.0.
3. Keep Iceberg's Avro version as 1.9.2 but try to avoid using classes that are not available in 1.8.2
4. De-bump Iceberg's Avro version to 1.8.2.

@electrum @rdblue @Fokko Could you folks take a look and comment?

---

FYI I tried to rebase #4776 but it depends on a fix in Iceberg 0.10.0-rc0 to pass some test cases in AbstractTestIcebergSmoke.

[electrum]

If we're going to update, should we use Avro 1.10.0?

I created a PR to test updating Avro: #5226

[lxynov]

Thanks @electrum ! FYI apache/hive#674 is a relevant PR

[rdblue]

Iceberg doesn't need to throw InvalidAvroMagicException. I just used it because it seemed appropriate, but I think I'd rather throw something else (ValidationException?) if it will avoid a compatibility problem. We can fix that before the 0.10.0 release.

[Fokko]

My preference would be definitely 1. Avro 1.8.2 is quite outdated as pointed out in https://issues.apache.org/jira/browse/HIVE-21737. Ismaël did a lot of work on it, but it got stuck by a test that was testing against an older version of Spark.

[lxynov]

Thanks @electrum for working on bumping presto-hive-apache's Avro version to 1.9.2. @rdblue @Fokko FYI it's a shaded Hive library that Presto uses.

@rdblue I believe avoiding using InvalidAvroMagicException does help resolve the compatibility issue. I did that in my forked repo, tested it with Presto, and Presto-Iceberg smoke tests passed. Just created a PR #5270 for comprehensive testing.

[electrum]

I just merged the change to update to Avro 1.9.2