Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump Avro version to 1.10.1 #8946

Closed
wants to merge 1 commit into from

Conversation

jirassimok
Copy link
Member

This should resolve #8943 and similar problems.

This fixes bugs where old Avro files/schemas couldn't be read.
@findepi
Copy link
Member

findepi commented Aug 23, 2021

@jirassimok see build outcome.

@hashhar
Copy link
Member

hashhar commented Aug 24, 2021

I consider apache/iceberg#1648 to be important (so version should be bumped with Iceberg version) since it looks like it can lead to silent loss of precision for decimal columns judging by:

The validation of downcasting decimals was made more strict. Therefore with the Parquet to Avro Decimal conversion, there was an implicit downcast from 34 to 10. The current conversion only takes scale into account, so it could be that precision would be lost.

@findepi
Copy link
Member

findepi commented Aug 24, 2021

I consider apache/iceberg#1648 to be important (so version should be bumped with Iceberg version)

so we should be able to bump Avro version for Hive connector, but not for Iceberg just yet

@electrum
Copy link
Member

electrum commented Sep 3, 2021

The Avro dependency in Trino is not used for Hive or Iceberg. You can see the modules like Kafka, BigQuery, Pinot, that use the dependency. For Hive and Iceberg, Avro comes from the shaded Hive depencency.

A potential issue with updating Avro for Hive is that Hive has code like org.apache.hadoop.hive.serde2.avro.AvroSerDe that might not be compatible with newer versions. If that turns out to be the case, we can solve this for Iceberg by moving Avro out of the shaded Hive dependency, and then the Hive connector can use the older version and the Iceberg connector can use the new version.

@electrum
Copy link
Member

electrum commented Sep 3, 2021

I put up a PR, let's see if the tests pass: #9114

@jirassimok
Copy link
Member Author

jirassimok commented Sep 8, 2021

Closed in favor of #9114.

@jirassimok jirassimok closed this Sep 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Upgrade Avro version to 1.10.1
4 participants