Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ORC: Fix importing ORC files with float and double columns #3320

Closed
wants to merge 1 commit into from
Closed

ORC: Fix importing ORC files with float and double columns #3320

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Fix importing ORC files with float and double columns.
  • Loading branch information
@rdblue
rdblue committed Oct 19, 2021
commit cf5f83bdf330d6eb20d2a6e9e84b556785be4c4b
16 changes: 11 additions & 5 deletions orc/src/main/java/org/apache/iceberg/orc/OrcMetrics.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,11 +196,17 @@ private static Optional<ByteBuffer> fromOrcMin(Type type, ColumnStatistics colum
min = Math.toIntExact((long) min);
}
} else if (columnStats instanceof DoubleColumnStatistics) {
// since Orc includes NaN for upper/lower bounds of floating point columns, and we don't want this behavior,
// we have tracked metrics for such columns ourselves and thus do not need to rely on Orc's column statistics.
Preconditions.checkNotNull(fieldMetrics,
"[BUG] Float or double type columns should have metrics being tracked by Iceberg Orc writers");
min = fieldMetrics.lowerBound();
if (fieldMetrics != null) {
// since Orc includes NaN for upper/lower bounds of floating point columns, and we don't want this behavior,
// we have tracked metrics for such columns ourselves and thus do not need to rely on Orc's column statistics.
min = fieldMetrics.lowerBound();
} else {
// imported files will not have metrics that were tracked by Iceberg, so fall back to the file's metrics.
min = ((DoubleColumnStatistics) columnStats).getMinimum();
if (type.typeId() == Type.TypeID.FLOAT) {
min = ((Double) min).floatValue();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if we want to reassign min to negative infinite if the this value tracked by ORC is NaN

}
}
} else if (columnStats instanceof StringColumnStatistics) {
min = ((StringColumnStatistics) columnStats).getMinimum();
} else if (columnStats instanceof DecimalColumnStatistics) {
Expand Down