Spark case-sensitive option mode

[jinchengchenghh]

Description

Presto only supports case-sensitive mode, but spark has configuration spark.sql.caseSensitive with default value false which describes as Whether the query analyzer should be case sensitive or not. Default to case insensitive. It is highly discouraged to turn on case sensitive mode.

Can we support case-insensitive mode in velox?

As spark implementation, it will convert parquet read column name to lowercase and convert operator column name to lowercase.

Velox can add a new config named case_sensitive in HiveConfig, and set it in QueryContext, The default value will be true matches velox original logical. If set it to false, we will convert the column name to lowercase when converting parquet schema to velox schema RowType.
If user uses case-insensitive mode, user should use lower case column name to create PlanNode, or we can add a function to convert original PlanNode to case-insensitive PlanNode.

What do you think? @mbasmanova

Kindly look forward to your reply!

[mbasmanova]

As spark implementation, it will convert parquet read column name to lowercase and convert operator column name to lowercase.

Column names in the plan are controlled outside of Velox, hence, no change in Velox is needed I think.

Re: Parquet reader - Can you clarify a bit more about the desired behavior? Do we need the reader to read column FOO as foo or something else?

[jinchengchenghh]

Yes, we need the reader to read column FOO or fOO as foo.

[mbasmanova]

@Yuhta Jimmy, do you know if Parquet reader can handle column names in a case insensitive ways already? If not, how hard would it be to change it?

[Yuhta]

It should be doable, this is not specific to parquet though.

1. We generate output type and filters using all lower case names in connector (if the planner can ensure this, we do not need to do this in Velox)
2. For the readers we add an option to return all column names as lower case
3. If 1 is needed in Velox, we change the column names in the result vector back to its original names in plan

[jinchengchenghh]

For the 2, I have one draft PR oap-project#126, now it is just for parquet reader, we can expand it to other readers.

[Yuhta]

@jinchengchenghh The idea looks good. Maybe one suggestion: the option at reader level should be readAsLowerCase as that is the actual behavior. Also we may want to add some check in connector to check all field names in output type and filters are actually lower case.

[jinchengchenghh]

Thanks for your suggestion. I will create a PR to do as you say.