[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull of an ArrayType to true

[yhuai]

Always set containsNull = true when infer the schema of JSON datasets. If we set containsNull based on records we scanned, we may miss arrays with null values when we do sampling. Also, because future data can have arrays with null values, if we convert JSON data to parquet, always setting containsNull = true is a more robust way to go.

JIRA: https://issues.apache.org/jira/browse/SPARK-6052

[SparkQA]

Test build #28050 has started for PR 4806 at commit 05eab9d.

• This patch merges cleanly.

[SparkQA]

Test build #28050 has finished for PR 4806 at commit 05eab9d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28050/
Test PASSed.

[viirya]

@yhuai is this pr also dealing with one of the problems reported in #4729? Looks like it is. If so, can we solve the problem in the original pr? Or please suggest on it, instead of just making each sub-problem of it as new pr such as #4782 and this? It makes the original pr hard to manage and modify. Thanks!

[viirya]

Besides, I think that it is weird to manually set up the containsNull for JSON schema inference. Sampling should not be an issue because you can also argue that we may miss arrays with different column types.

So the main point is still the problem of inserting JSON data to parquet data source table. I did in #4729 just copy the schema of JSON data and modify its containsNull then use it for insertion, without actually modifying the schema of the JSON data.

Both solutions are working on the unit test. @liancheng @yhuai you can decide which one is more proper.

[liancheng]

@viirya Making complex types in JSON relation always nullable could be more robust, and makes more sense for most common use cases. We don't want to get a schema with wrong nullability when we happened to only sampled records without nulls. So this problem itself should be fixed anyway regardless of SPARK-5950.

[liancheng]

Merging to master and branch-1.3, thanks!

[viirya]

I think your suggestion to completely remove nullablity may be considered if it is useless at all.

[liancheng]

Yeah, we introduced it for potential optimizations, but seems that it's causing more troubles. We decided to ignore nullability in Parquet and JSON data sources because this seems to be making more sense for most scenarios, especially when dealing with "dirty" datasets.

However, completely ignoring nullability in Spark SQL also means that we lose part of the schema information, which affects data sources like Avro, ProtocolBuffer, and Thrift. Not quite sure whether this is a good idea for now...