[SPARK-25132][SQL][BACKPORT-2.3] Case-insensitive field resolution when reading from Parquet

[seancxmao]

What changes were proposed in this pull request?

This is a backport of #22148

Spark SQL returns NULL for a column whose Hive metastore schema and Parquet schema are in different letter cases, regardless of spark.sql.caseSensitive set to true or false. This PR aims to add case-insensitive field resolution for ParquetFileFormat.

• Do case-insensitive resolution only if Spark is in case-insensitive mode.
• Field resolution should fail if there is ambiguity, i.e. more than one field is matched.

How was this patch tested?

Unit tests added.

[gatorsmile]

See my comment: https://github.com/apache/spark/pull/22184/files#r212006137

[cloud-fan]

ok to test

[cloud-fan]

+1 to backport this. I think it's a bug that we don't respect case-sensitive config when resolving parquet fields.

[yucai]

We need to backport it. Without this PR, we cannot solve the data issue in [SPARK-25206] Wrong data may be returned when enable pushdown.

[SparkQA]

Test build #95215 has finished for PR 22183 at commit 2831588.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

For Hive tables, column resolution is always case insensitive. However, When spark.sql.hive.convertMetastoreParquet is true, users might face inconsistent behaviors when they use native parquet reader to resolve the columns in the case sensitive mode. We still introduce behavior changes. Better error messages sounds good enough, instead of disabling spark.sql.hive.convertMetastoreParquet when the mode is case sensitive.

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

BTW, @gatorsmile and @cloud-fan, do you know who did this ^ and why?

[cloud-fan]

As discussed in the JIRA, this is a partial fix, and we need to backport another 2 PRs, which is risky. Can we close it?

[HyukjinKwon]

ohh.. no no .. I meant:

Can one of the admins verify this patch?

[cloud-fan]

ah, I think there is a service that will comment on stale PRs to ask people to review. I don't who maintain this service though...