Skip to content

Commit

Permalink
[Bug]Parquet map/list/struct structure recognize (apache#4968)
Browse files Browse the repository at this point in the history
When a parquet file contains a `Map/List/Struct` structure, Doris can not recognize the column correctly,
and throws exception 'Invalid column: xxxx', that means Doris can not find the column.
The `Map` structure will be recognized into two columns: `key and value`.
The follow is the schema of a parquet file recognized by Doris. This patch tries to solve this problem.
  • Loading branch information
xinghuayu007 authored Nov 28, 2020
1 parent cb749ce commit 2331ce1
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion be/src/exec/parquet_reader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,11 @@ Status ParquetReaderWrap::init_parquet_reader(const std::vector<SlotDescriptor*>
auto *schemaDescriptor = _file_metadata->schema();
for (int i = 0; i < _file_metadata->num_columns(); ++i) {
// Get the Column Reader for the boolean column
_map_column.emplace(schemaDescriptor->Column(i)->name(), i);
if (schemaDescriptor->Column(i)->max_definition_level() > 1) {
_map_column.emplace(schemaDescriptor->Column(i)->path()->ToDotVector()[0], i);
} else {
_map_column.emplace(schemaDescriptor->Column(i)->name(), i);
}
}

_timezone = timezone;
Expand Down

0 comments on commit 2331ce1

Please sign in to comment.