Parquet: Fix value def level when max def level is 0

When running with a release build, NULL would be returned when
reading values from required fields in parquet files (with a debug
build a DCHECK would be hit).

Previously when the max definition level for a field was 0 (which
happens if a field is required), the definition level for value was
incorrectly set to 1. The max definition level is related to nested
data and is defined to be the number of nullable fields that will be
encountered when traversing a path to reach the desired end field.
For example, if a nested schema has a path a.b.c.d where b and d are
nullable then the max def level is 2. A def level is attached to each
value to indicate the number of optional values that are present (in
the previous example an def level of 2 means both b and d are not
null). So having a def level for a value that is greater than the max
def level for a field should never happen.

Change-Id: Ia91a97cf79e672c420d10416c6817f0930dcc920

Author: Casey Ching

be/src/exec/hdfs-parquet-scanner.cc

@@ -719,7 +719,7 @@ inline int HdfsParquetScanner::BaseColumnReader::ReadDefinitionLevel() {
   if (max_def_level() == 0) {
     // This column and any containing structs are required so there is nothing encoded for
     // the definition levels.
-    return 1;
+    return 0;
   }
 
   uint8_t definition_level;

testdata/data/kite_required_fields.parquet

@@ -1338,6 +1338,28 @@ ${IMPALA_HOME}/testdata/data/bad_compressed_size.parquet \
 /test-warehouse/bad_compressed_size_parquet/
 ====
 ---- DATASET
+-- Parquet file with required columns written by Kite. Hive and Impala always write files
+-- with fields as optional.
+functional
+---- BASE_TABLE_NAME
+kite_required_fields
+---- COLUMNS
+req_int bigint
+opt_int bigint
+req_string string
+opt_string string
+req_bool boolean
+opt_bool boolean
+opt_int_2 bigint
+opt_int_3 bigint
+req_int_2 bigint
+req_int_3 bigint
+---- LOAD
+`hadoop fs -mkdir -p /test-warehouse/kite_required_fields_parquet && \
+hadoop fs -put -f ${IMPALA_HOME}/testdata/data/kite_required_fields.parquet \
+/test-warehouse/kite_required_fields_parquet/
+====
+---- DATASET
 functional
 ---- BASE_TABLE_NAME
 bad_serde

testdata/datasets/functional/functional_schema_template.sql

@@ -40,6 +40,7 @@ table_name:bad_metadata_len, constraint:restrict_to, table_format:parquet/none/n
 table_name:bad_dict_page_offset, constraint:restrict_to, table_format:parquet/none/none
 table_name:bad_compressed_size, constraint:restrict_to, table_format:parquet/none/none
 table_name:alltypesagg_hive_13_1, constraint:restrict_to, table_format:parquet/none/none
+table_name:kite_required_fields, constraint:restrict_to, table_format:parquet/none/none
 
 # TODO: Support Avro. Data loading currently fails for Avro because complex types
 # cannot be converted to the corresponding Avro types yet.

testdata/datasets/functional/schema_constraints.csv

@@ -36,3 +36,13 @@ SELECT * from bad_compressed_size
 ---- CATCH
 Column 0 has invalid column offsets (offset=4, size=1000000, file_size=245)
 ====
+---- QUERY
+# Parquet file with required fields.
+select * from kite_required_fields
+---- TYPES
+bigint,bigint,string,string,boolean,boolean,bigint,bigint,bigint,bigint
+---- RESULTS
+1,2,'foo','bar',true,false,1,2,3,4
+1,NULL,'foo','NULL',true,NULL,NULL,NULL,3,4
+100,NULL,'foooo','NULL',false,NULL,NULL,NULL,300,400
+====