[SPARK-10310] [SQL] Fixes script transformation field/line delimiters

[liancheng]

Please attribute this PR to Zhichao Li <zhichao.li@intel.com>.

This PR is based on PR #8476 authored by @zhichao-li. It fixes SPARK-10310 by adding field delimiter SerDe property to the default LazySimpleSerDe, and enabling default record reader/writer classes.

Currently, we only support LazySimpleSerDe, used together with TextRecordReader and TextRecordWriter, and don't support customizing record reader/writer using RECORDREADER/RECORDWRITER clauses. This should be addressed in separate PR(s).

[SparkQA]

Test build #42803 has finished for PR 8860 at commit 7c4b03b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhichao-li]

@liancheng I guess there's still some issue like:
i.e: If user specify serde with "LazySimpleSerde" explicitly, then it would use Text.write to serialize which would not match the behavior of tab as field delimeter and \n as the line delimiter.
Maybe we can adding more support in the other PR since most of the user would only depend on the default behavior, and this can cover the majority usages. :)

[liancheng]

@zhichao-li I further special cased LazySimpleSerDe, so that we always use TextRecordReader/TextRecordWriter together with it, and uses can customize field delimiter now. Please check this test case.

[yhuai]

test this please

[SparkQA]

Test build #42846 has finished for PR 8860 at commit f86b5bc.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42846/testReport/junit/org.apache.spark.sql.hive.execution/SQLQuerySuite/SPARK_10310__script_transformation_using_LazySimpleSerDe/

Looks like a legitimate failure?

[SparkQA]

Test build #42848 has finished for PR 8860 at commit 387ac72.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #42855 has finished for PR 8860 at commit 387ac72.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CountVectorizer(JavaEstimator, HasInputCol, HasOutputCol):
  • class CountVectorizerModel(JavaModel):
  • s"Failed to convert value $v (class of $
  • s"Failed to convert value $v (class of $
  • case class Sort(

[yhuai]

@zhichao-li Can you try this PR?

[zhichao-li]

LGTM

[yhuai]

Thanks! Merging to master and branch 1.5.